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Executive  Summary 


This  report  documents  the  findings  and  results  of  Phases  1  and  2  of  RT40,  Quantitative  Technical  Risk. 
The  Quantitative  Technical  Risk  project  is  envisioned  as  a  multi-year  effort  to  develop  and  transition 
effective  methods,  procedures  and  tools  (MPT)  to  improve  the  effectiveness  of  risk  management  in  DoD 
system  development  programs. 

Phase  1  emphasized  foundational  research  for  a  framework  and  approach  to  examine  representations 
of  the  evolving  system  design  to  locate  sources  of  complexity  and  quantify  system  complexity  in  order  to 
provide  insight  into  development  and  performance  risk.  Phase  1  ran  from  November  2012  through 
August  2013,  and  is  documented  in  Volume  I  of  this  report. 

The  project  adjusted  emphasis  between  Phase  1  and  Phase  2.  The  motivation  for  the  change  in 
emphasis  were  the  observations  that  (a)  risk  assessment  based  on  analysis  system  complexity  required  a 
level  of  design  detail  that  is  not  available  until  too  late  in  the  acquisition  lifecycle  to  inform  the  system 
acquisition  system  decision  making,  and  (b)  system  complexity  may  be  only  one  of  many  sources  of  risk. 
This  change  in  focus  was  accompanied  by  a  change  in  strategy  from  foundational  research  to  one  of 
incremental  development,  testing  and  transition  to  end-users  of  relevant  and  practical  near-term  MPT 
that  would  become  the  core  of  a  broader  toolset. 

Phase  2  emphasized  developing  a  framework  to  expand  the  traditional  DoD  characterization  of  risk,  and 
an  approach  that  integrated  risk  management  with  the  sequence  and  structure  of  program  decisions  at 
different  stages  of  acquisition  and  program  components.  Risks,  in  the  traditional  DoD  characterization 
are  specific  future  events  that  have  some  probability  and  consequence  that  can  be  estimated  in 
advance.  The  expanded  characterization,  modeled  after  risk  analysis  in  other  domains,  profile  the  risk 
propensity  and  risk  exposure  of  the  program  as  measured  by  a  set  of  risk  indicators.  Decision  making  is 
informed  by  assessing  the  impact  alternative  decision  choices  on  the  risk  profile.  Warning  thresholds  on 
the  risk  indicators  alert  program  management  for  the  need  to  consider  risk  mitigation.  Characteristic 
program  risks,  indicators,  and  potential  mitigations  are  linked  in  a  Risk  Breakdown  Structure  (RBS).  The 
RBS  is  generated  by  applying  Process  Failure  Modes  and  Effects  Analysis  (P-FMEA)  to  the  program 
decision  breakdown  structure.  Phase  2  ran  from  September  2013  to  12  November  2013,  and  is 
documented  in  Volume  II  of  this  report. 

The  following  phases  will  incrementally  develop,  pilot  test,  refine,  enhance  and  transition  specific  MPT, 
in  collaboration  with  end-user  co-developer  transition  partners. 
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Volume  I  -  Phase  1  (November  2012  to  August  2013) 


Research  Team 

Stevens  Institute  of  Technology:  Dr.  Roshanak  Nilchiani,  Dr.  Stan  Rifkin,  Dr.  Ali  Mostashari, 
Wayne  State  University:  Dr.  Walt  Bryzik,  Dr.  Gary  Witus, 


Abstract 

A  novel  approach  to  technical  risk  identification  and  analysis  for  major  weapons  systems  acquisitions  is 
proposed.  It  is  informed  by  the  limitations  of  the  current  risk  matrix.  The  approach  is  to  examine 
representations  of  the  evolving  system  design  to  locate  sources  of  complexity  and  then  inform  the 
program  manager  as  he/she  makes  technical  choices  among  competing  alternatives.  Some  of  the 
alternatives  will  create  more  complexity  and  therefore  more  risk.  The  PM  will  then  be  able  to  balance 
risk  and  reward  at  the  point  of  decision-making,  deciding  to  engage  risk  at  that  moment  by  his/her 
choices.  In  addition,  we  propose  to  rate  or  score  the  contractor  +  government  organizations'  abilities  to 
master  the  complexity  they  have  chosen,  so  that  the  PM  will  know  whether  there  is  a  match  of  product 
complexity  with  organizational  capability. 

Future  work  will  add  dimensions  of  interconnections  and  interdependencies  among  risks,  timing,  delay, 
order  of  risks,  and  uncertainty. 
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Challenges 


"It  is  not  possible  to  know  exactly  how  a  particular  design  will  perform  until  it  is  built.  But  the  product 
cannot  be  built  until  the  design  is  selected.  Thus,  design  is  always  a  matter  of  decision  making  under 
conditions  of  uncertainty  and  risk"  [(Hazelrigg,  1998),  quoted  in  (Deshmukh,  2010),  p.  128]. 


1.  Risk  management  is  many  processes 

Defined  in  the  DoD  Risk  Management  Guide  (2006,  p.  1), 

Risk  is  a  measure  of  future  uncertainties  in  achieving  program  performance  goals  and  objectives 
within  defined  cost,  schedule  and  performance  constraints.  Risk  can  be  associated  with  all 
aspects  of  a  program  (e.g.,  threat,  technology  maturity,  supplier  capability,  design  maturation, 
performance  against  plan,) ....  Risk  addresses  the  potential  variation  in  the  planned  approach 
and  its  expected  outcome. 


Risks  have  three  components: 


•  A  future  root  cause  (yet  to  happen),  which,  if  eliminated  or  corrected,  would  prevent  a 
potential  consequence  from  occurring, 

•  A  probability  (or  likelihood)  assessed  at  the  present  time  of  that  future  root  cause  occurring, 
and 

•  The  consequence  (or  effect)  of  that  future  occurrence. 

A  future  root  cause  is  the  most  basic  reason  for  the  presence  of  a  risk.  Accordingly,  risks  should 
be  tied  to  future  root  causes  and  their  effects. 


Further  risk  management  is  a  number  of  processes: 
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Figure  1.  DoD  Risk  Management  Process  (from  DoD  Risk  Management  Guide,  2006,  p.  4) 

Of  these  processes,  which  are  the  most  important  to  practice,  to  "get  right"?  If  we  want  to  improve  the 
management  of  technical  risks,  on  which  process(es)  should  we  focus? 

Our  conjecture  is  that  Risk  Identification  and  Risk  Analysis  are  key  because,  as  in  the  Figure,  everything 
else  depends  upon  them.  So,  we  are  starting  there. 


2.  Forecasting  risk  is  difficult  and  subjective 

As  listed  above,  program  management  needs  to  collect  and  identify:  future  root  causes,  likelihood,  and 
consequences.  This  is  often,  under  the  best  circumstances,  by  assembling  experts  and  asking  them  to 
converge  on  the  three  components.  It  is  a  group  process  and  based  on  the  extensive  practical 
experience  of  the  expert  panel.  It  is  necessarily  subjective,  based  on  the  memories  of  the  panel 
members  and  their  analogic  reasoning. 

Tversky  and  Kahneman  (see,  for  example,  (Kahneman,  Slovic,  &  Tversky,  1982))  are  celebrated  for  their 
prospect  theory,  which  explains  how  our  biases  interfere  with  our  rational  appraisals.  They  explain  how 
our  subjective  judgments  are  susceptible  to  internal  and  also  normative  forces  that  cloud  our 
perceptions  and  our  reasoning.  Accordingly,  subjective  assessments  of  technical  risk  are  vulnerable  to 
biases. 

And  this  mentions  nothing  about  the  problem  of  using  analogic  reasoning,  which  is  what  expert  team 
members  often  apply.  The  challenge  in  analogical  reasoning  is  that  the  strength  of  the  relevant 
similarities  must  outweigh  the  strength  of  any  significant  dissimilarity.  But  is  that  what  happens  when 
experts  get  together  to  evaluate  risks?  When  one  expert  says,  "This  is  just  like  Project  X,  so  I  can  foresee 
a  risk  of  type  A,"  is  the  analogy  apt?  Is  it  questioned?  [Here  is  a  list  with  examples  of  errors  by  analogy: 
http://www.skepdic.com/falseanalogy.html] 
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Expert  panels,  in  creating  their  subjective  assessments,  are  susceptible  to  both  biases  and  errors  in 
analogical  reasoning. 


3.  Risk  identification  is  almost  always  post  hoc 

One  part  of  the  Risk  Management  Guide  states,  "Use  a  proactive,  structured  risk  assessment  and 
analysis  activity  to  identify  and  analyze  root  causes, "  (p.  5)  and  others  state,  "Use  the  results  of  prior 
event-based  systems  engineering  technical  reviews  to  analyze  risks  potentially  associated  with  the 
successful  completion  of  an  upcoming  review,"  (p.  5)  and  "During  decomposition,  risks  can  be  identified 
based  on  prior  experience,  brainstorming,  lessons  learned  from  similar  programs,  and  guidance 
contained  in  the  program  office  RMP  [Risk  Management  Plan]."  p.  7. 

Looking  backwards  to  find  risks  is  unassailable,  except  that  it  is  applied  by  judgment  and  analogy  and 
therefore  subject  to  bias  and  error.  For  all  of  the  looking  backwards,  the  purpose  is  to  predict  the  future. 
Where  is  the  justification  of  the  predictions? 


Our  Approach  and  Its  J  ustification 


Summary 

Our  approach  is  to  characterize  aspects  of  the  technical  products  being  developed  in  a  way  that  would 
inform  a  program  manager  about  to  make  decisions  by  weighing  alternative  technical  courses  of  action. 
We  would  score  or  rank  the  alternatives  based  on  the  relationship  that  each  alternative  would  incur 
future  risk. 


The  result  of  our  research  would  be  a  scorecard,  dash  board,  or  workbench  that  the  program  office 
operates  before  each  major  technical  decision.  The  workbench  would  be  fed  information  about  the 
nature  of  the  product  alternatives  and  based  on  that  would  compute  the  relative  attractiveness  of  each 
option  with  respect  to  incurring  future  risk. 


In  addition  to  examining  characteristics  of  the  product,  the  workbench  would  also  scrutinize  the  match 
between  the  characteristics  of  the  product  and  the  characteristics  of  the  developing  organization,  as  risk 
can  arise  relative  to  an  organization's  capability  to  develop  a  certain  product  alternative. 


As  our  research  progresses,  we  intend  to  add  capabilities  to  take  into  account  the  order  and  timing  of 
decisions,  their  cascade  effects,  and  the  impact/influence  of  uncertainty. 
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We  are  taking  this  approach  for  a  number  of  reasons: 

1.  The  current  method  of  risk  characterization,  measurement,  and  mitigation  has  not  improved 
even  though  the  Department  of  Defense  has  spent  tens  of  millions  of  dollars  on  research  to 
improve  it.  Evidently  the  research  results  have  not  proven  useful  enough  to  change  the 
guidance.  After  all,  much  of  the  DoD  research  investment  has  been  in  Bayesian  methods,  which 
have  been  around  for  almost  200  years  and  still  have  not  found  their  way  into  the  published 
guidance. 

2.  We  have  all  heard  the  remark,  usually  made  informally  by  those  who  see  many  major 
weapons  systems  acquisitions,  that  by  the  time  the  real  issues  become  visible  it  is  very  late  and 
the  effects  have  spread.  We  seek  to  identify  risky  courses  of  action  at  the  time  they  are  being 
considered  for  selection.  This  is  very  early  in  the  unfolding  of  the  systems  development, 
hopefully  in  time  to  take  alternative  steps  if  unaddressed  impacts  are  discovered. 


A.  Objective  Assessment 

Our  approach  does  not  use  any  judgment,  only  objective  measures  of  the  product  and  of  the 
organization's  capability  to  create  the  product.  Accordingly,  we  hope  to  circumvent  the  subjective  biases 
that  can  be  found  in  the  current  DoD  risk  identification  and  analysis  practices. 


B.  Quantifiable  assessment 

We  seek  to  compute  characteristics  about  the  product  alternatives  and  about  organizational  capability, 
so  the  outputs  of  our  analysis  would  be  quantities  that  would  aid  program  management  in  making 
decisions  among  competing  technical  alternatives. 


C.  Aid  in  Decision  Making 

Since  our  approach  is  a  tool  to  be  used  during  decision-making,  we  are  not  taking  a  retrospective  view 
per  se,  but  rather  trying  to  give  the  PM  information  in  order  to  avoid  risk,  that  is,  avoid  encountering  a 
state  of  nature  that  potentially  would  have  unacceptably  high  likelihood  and  consequence. 


D.  Time  is  a  variable,  risks  are  interconnected 

There  is  no  explicit  time  dimension  in  the  current  DoD  risk  management  practice.  We,  on  the  other 
hand,  see  technical  risks  as  largely  interdependent/interconnected,  so  the  order  in  which  the  technical 
decisions  are  considered  matters.  Accordingly,  as  our  research  progresses  we  intend  to  be  able  to 
present  a  program  manager  with  the  options  during  decision-making  of  understanding  the  effects  of 
deferring  or  accelerating  certain  technical  decisions. 


In  addition,  time  plays  another  important  role  in  risk  because  time  delay  between  cause  and  effect 
interferes  with  our  ability  to  connect  the  two,  our  ability  to  reason  about  what  the  root  causes  are  of 
untoward  and/or  unexpected  program  outcomes.  Therefore,  characterizing  the  time-dependent  (that  is. 
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dynamic)  flow  through  the  program  and  technical  product  structures  is  crucial  to  identifying  real,  latent 
causes,  not  just  their  surface  symptoms,  such  as  cost  and  schedule  over-runs. 

E.  Advances  in  Risk  mangement:  Where  to  Look 

A  great  deal  of  work  already  has  been  done  on  improvements  to  risk  identification  and  risk  analysis.  For 
example,  the  DoD  has  sponsored: 

The  Software  Engineering  Institute's  Risk  Program  for  several  decades. 

University  of  Virginia's  Center  for  Risk  Management  of  Engineered  Systems  for  several  decades. 
Research  at  the  Air  Force  Institute  of  Technology  and  Naval  Postgraduate  School  for  decades. 
Research  and  application  at  its  FFRDCs,  such  as  MITRE  and  Aerospace,  for  decades. 


While  the  knowledge  created  at  those  institutions  has  varied,  much  of  it  centered  on  obtaining  a  more 
complete  list  of  risks  and  better  estimates  of  the  likelihood  and  consequence.  Evidently  the  fruits  have 
not  been  powerful  enough  to  change  the  written  DoD  guidance. 


One  could  consult  the  major  defense  contractors,  as  for  decades  they  have  been  actively  managing  the 
risks  of  developing  weapons  systems.  We  approached  a  few  of  them  informally  to  ask  if  they  would 
discuss  with  us  their  risk  management  methods.  They  responded  that  they  considered  their  risk 
management  practices  to  be  competition  sensitive  and  determinative  of  their  commercial  success  and 
would  not  share  them.  We  also  approached  a  few  industrial  firms  and  received  the  same  answer. 


What  about  firms  that  deal  in  risk  every  day,  such  as  insurance  and  investment  businesses?  Flere  the 
final  report  of  a  previous  SERC  research  topic,  valuing  flexibility  (RT-18),  is  dispositive  (Deshmukh,  2010). 


But  what  is  the  connection  between  valuing  flexibility  and  risk?  One  parallel  is  that  both  attempt  to 
characterize  future  uncertainties.  After  all,  flexibility  is  about  responses  to  future  changes,  some 
unplanned.  "Most  approaches  for  valuing  flexibility  depend  on  good  estimation  of  uncertainty. 
Flowever,  estimating  and  characterizing  uncertainty,  even  for  foreseeable  sources  of  [change],  is 
difficult,  especially  in  systems  involving  new  technologies."  p.  24 


Investment  advisors  often  use  the  technique  of  Real  Options  to  find  the  best  investment  among 
alternatives,  akin  to  what  acquisition  program  managers  must  do  at  multiple  points  during 
development.  Flere  are  some  weaknesses  of  Real  Options  in  the  DoD  context  (p.  62): 


"  Financial  options'  assumptions,  such  as  no  arbitrage  condition,  complete  market  condition  and 
infinite  liquidity,  may  not  hold  for  the  non-financial  market. 
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"Without  checking  the  assumption  of  Black-Scholes  model,  using  the  Black-  Scholes  formula  does 
not  make  sense.  For  example,  the  strongest  assumption  of  the  model  is  the  fact  that  uncertainty  can 
be  modeled  in  geometric  Brownian  motion  and  as  a  result  the  distribution  of  future  status  is  [a]  log¬ 
normal  distribution.  If  the  future  environment  cannot  be  modeled  with  this  stochastic  process  and 
distribution,  the  Black-Scholes  model  is  not  valid. 


"Almost  all  real  options  related  literature  assumes  the  risk-neutral  decision  maker  implicitly  or 
explicitly.  This  assumption  need[s]  to  be  check[ed]  in  [the]  risk  management  sense." 


Further,  the  report  continues,  with  some  overlap  with  the  previous  list  (p.  64): 


"  Real  options  must  be  described  in  terms  of  specific  technologies  and  the  systemic  domain  in  which 
they  are  to  be  developed.  Financial  analysis  alone  is  insufficient  to  frame  real  options.  This  is  quite 
difficult,  when  as  yet  undeveloped  technologies  are  under  consideration. 


"Financial  options  are  well-defined  contracts  that  are  tradable  and  individually  valued,  while  real 
options  are  not:  real  options  have  no  contract-specified  exercise  price  of  time.  The  usefulness  of 
valuing  every  potential  program  alternative  that  provides  flexibility  is  not  clear. 


"In  military  procurement  programs,  previous  experiences  associated  with  the  development  of 
similar  technologies  are  not  necessarily  available.  Flence,  valuing  real  options  on  the  basis  of  so 
called  "comparables"  becomes  questionable  because  of  the  absence  of  available  data. 


"Real  options  are  most  often  path-dependent.  Flence,  direct  applicability  of  traditional  financial 
options  methodologies  is  not  appropriate,  as  the  underlying  stochastic  differential  equations  are  not 
necessarily  available. 


"Real  options  in  military  acquisition  programs  are  likely  to  be  highly  interdependent.  Traditional 
financial  option  pricing  methods  fail  here,  again,  because  underlying  stochastic  differential 
equations  may  be  unattainable. 


"In  military  procurement  programs,  there  may  be  no  justifiable  reason  to  accept  the  "no  arbitrage 
assumption".  In  this  case,  general  option  pricing  theory  breaks  down. 


"There  is  typically  no  quantitative  or  qualitative  reason  to  believe  the  real  options  have  uncertainty 
in  price  that  follow  Brownian  motion.  That  is,  unlike  in  financial  markets  where  there  exist  both 
quantitative  and  qualitative  analyses  that  support  by  weak  convergence  in  measure  principles  that 
suggests  a  limiting  Brownian  motion  price  process,  there  is  typically  no  similar  reasoning  supporting 
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the  assumption  of  Brownian  behavior.  Hence,  the  semi-martingale  arguments  leading  to  the 
principal  results  of  general  option  pricing  are  not  applicable." 


And  this  does  not  even  address  what  may  be  the  most  difficult  part  of  the  application  of  Real  Options  in 
the  DoD  context:  the  necessity  to  assess  the  probability  of  each  state  of  nature  in  the  unfolding  of  future 
events.  Investment  analysts  use  historical  information  to  estimate  those  probabilities,  but  there  is  little 
on  which  to  base  estimates  of  weapons  systems  development  probabilities,  especially  of  new 
capabilities. 


In  the  end,  we  cannot  rationally  defend  what  some  other  communities,  above,  use  to  manage  risk 
because  their  assumptions  and  sources  of  data  match  so  little  of  our  situation. 
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Connecting  technical  risk  and  types  of  complexity 


A.  A  FEW  DEFINITIONS 

The  field  of  complexity  is  rich  and  spans  over  the  past  half  century  in  various  fields  of  knowledge  ranging 
from  biological  systems  to  cyber-physical  systems.  As  it  has  been  discussed  by  several  researchers,  an 
strong  correlation  can  be  observed  between  the  complexity  of  the  system  and  various  ranges  of  failures, 
including  catastrophic  failures  (Merry,  1995;  Cook,  2000,  Bar-Yam,  2003). 


The  term  "complexity"  has  several  definitions  and  various  related  aspects  and  characteristics  in  various 
domains  of  knowledge.  We  adopt  the  following  definition: 


Complexity  is  the  potential  of  the  system  to  exhibit  unexpected  behavior  (Wiiicox,  2011) 


Complex  systems  exhibit  the  potential  for  unexpected  behavior  with  respect  to  variables  of  interest.  The 
potential  can  manifest  itself  in  certain  situations  and  create  the  actual  emergent  behavior  or  stay  hidden 
as  a  potential.  Complex  systems  have  non-linear  interactions,  circular  causality  and  feedback  loops.  They 
may  harbor  logical  paradoxes  and  strange  loops.  Small  changes  in  a  part  of  a  complex  system  may  lead 
to  emergence  and  unpredictable  behavior  in  the  system  (Erdi,  2008).  It  should  be  noted  that  complex 
systems  are  very  different  from  complicated  systems,  and  there  is  a  tendency  for  mistake  in  using  these 
terms  interchangeably.  Complicated  systems  often  have  many  parts,  however  the  interactions  between 
parts  and  subsystems  are  often  well  known  and  linear,  so  they  do  not  show  emergent  or  non-linear 
behavior.  In  contrast,  complex  system  may  or  may  not  have  many  parts,  however,  at  least  one  non¬ 
linear  behavior  of  feedback  loop  exists  in  the  structure  of  the  system  that  drives  emergence  and 
unknown  unknowns  in  the  system. 

The  increased  complexity  is  often  associated  with  increased  fragility  and  vulnerability  of  the  system.  By 
harboring  an  increased  potential  for  unknown  unknowns  and  emergent  behavior,  the  probability  of 
known  interactions  that  lead  to  performance  and  behavior  in  a  complex  system  decreases,  which  in  turn 
leads  to  a  more  fragile  and  vulnerable  system.  That  is,  the  presence  of  complexity  in  a  system,  even  a 
little  complexity,  can  swamp  the  behavior  of  the  familiar,  linear  interactions. 
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Figure  2.  Map  of  the  leading  scholars  and  areas  of  research  in  the  complexity  sciences 
(http://en.wikipedia.org/wiki/Complexity). 


As  can  be  seen  (but  may  not  be  able  to  read!)  from  this  figure,  there  are  many  threads  of  research  and 
many  definitions  of  complexity.  Our  job  is  to  pick  and  choose  among  the  relevant  threads  of  research 
which  can  contribute  to  the  understanding  of  complexity  at  various  milestones  of  acquisition  programs, 
and  identify  the  ones  most  applicable  to  characterizing  technical  risk. 


At  this  early  juncture,  we  can  say  only  that  we  have  not  focused  on  the  following  areas  in  the  diagram 
because  we  think  they  are  not  relevant  to  acquisition  programs: 

Artificial  intelligence  (distributed  neural  networking) 

Agent-based  modeling  (cellular  automata,  genetic  algorithms,  artificial  life,  multi-agent 
modeling 

Case-based  modeling 
New  science  of  networks 
Global  network  society 
Fractal  geometry 

Synergetics/macroscopic  modeling 
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Ecological  systems  theory 
Fuzzy  logic 


We  are  selectively  making  our  way  through  the  remainder  to  assess  suitability  to  characterize  technical 
risk  based  on  what  the  government  sees  during  the  acquisition  life  cycle.  Certain  areas,  such  as 
emergence,  are  potent  metaphors,  but  there  is  a  connotation  among  complexity  researchers  that 
emergence  is  a  property  that  cannot  be  sensed  by  looking  at  components,  so  for  the  moment  we  are 
not  investigating  emergence  further. 


We  are  looking  closely  at  these  kinds  of  complexity,  in  particular: 


•  Structural  -The  arrangement  of  pieces  and  parts  that  has  loops,  circuits,  so  that  feedback  is 
possible. 

•  Dynamic  -  The  behavior  of  a  system  that  unfolds  as  it  executes.  Here  we  look  for  delays  and 
non-linearities. 

•  Interface,  interconnection  -  How  parts  communicate  and  touch  each  other  and  whether  that 
connection  is  across  a  barrier,  whether  there  is  a  tight  or  loose  connection,  whether  information 
is  hidden  inside  the  components,  and  whether  the  parts  are  of  different  "kinds." 


B.  How  COMPLEXITY  MANIFESTS  AS  TECHNICAL  RISK 

B.1  Mechanisms  AND  examples 

One  example  of  risk  is  interconnecting  inhomogeneous  elements.  The  term  is  meant  broadly,  as  it  could 
refer  to  trying  to  connect  two  systems  that  had  never  been  connected  before,  even  though  each  of 
them  was  mature  in  itself.  The  poster-child  for  this  type  of  risk  is  DIVAD,  the  M247  Sergeant  York  "self- 
propelled  anti-aircraft  gun"  (en.wikipedia.org/wiki/DIVAD).  Due  to  the  urgent  need  for  the  capability,  a 
decision  was  made  by  the  Army  to  select  a  design  that  joined  three  commercial  off  the  shelf  systems:  an 
Army  M48  Patton  tank  chassis,  a  radar,  and  a  cannon. 

The  three  particular  commercially  off  the  shelf  systems  selected  by  the  vendor  had  never  been 
connected  and  the  computer  control  system  at  the  heart  had  not  yet  been  developed.  In  the  end,  the 
tank  was  too  slow  to  protect  the  ground  vehicles  it  was  intended  to.  The  radar,  while  off  the  shelf,  was 
off  the  shelf  for  an  airplane!  Airplane  radars  work  internally  by  detecting  movement.  Clearly,  a  tank  in 
the  field  was  not  (always)  in  motion  and  nor  were  its  targets.  The  physical  layout  of  the  radar  with 
respect  to  the  cannon  had  the  cannon  sometimes  getting  into  the  radar's  line  of  sight.  The  tank's  turret 
moved  too  slowly  to  track  realistic  air  targets  because,  after  all,  it  was  never  meant  to.  The  list  went  on. 
And  the  program,  comprised  of  commercial  off  the  shelf  systems,  was  cancelled. 
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How  would  our  analysis  have  identified  these  risks?  By  looking  for  inhomogeneous  interfaces. 


A  second  type  of  complexity  comes  from  feedback  and  delay.  Feedback  itself  is  a  structural 
characteristic:  it  is  a  loop  somewhere  in  the  product  being  developed  or  in  the  organization  creating  the 
product.  And  the  loop  can  amplify  or  dampen  the  signal  passing  through  it,  distorting  the  original  (think 
of  the  child's  game  of  "telephone").  And  the  transit  may  be  delayed  at  points,  which  creates  difficulty  for 
us  humans  to  reason  about  what  causes  the  effects,  the  surface  symptoms,  that  we  see. 


The  field  of  system  dynamics  is  awash  in  examples  of  loops  and  delays,  and  there  is  even  something  of  a 
cottage  industry  in  one  particular  example,  the  Beer  Game\  in  which  a  single  instance  of  a  change  in  a 
single  signal  causes  the  humans  operating  the  game  to  respond  in  a  way  that  causes  oscillation  that 
appears  to  be  unable  to  be  dampened.  All  of  this  due  to  the  (underlying)  structure  of  the  system, 
illustrating  that  structure  produces  behavior. 


The  example  below  comes  from  a  book  on  business  management  (Beer,  1979),  written  to  create  interest 
in  cybernetics.  In  this  example  are  trying  to  construct  a  system  that  has  even  an  output  around  the  value 
0,  given  an  input  single  in  the  form  of  a  regular  sine  wave: 


Figure  3.  An  output  varying  regularly  about  a  mean  value  that  is  its  target,  showing  the  corrections  at  appear 
necessary  at  each  time  epoch  when  the  measurement  is  made.  (Beer,  1979),  p.  60 


The  approach  is  to  generate  a  -1  when  we  see  a  +1  and  generate  a  +1  when  we  see  a  -1.  Here  is  what 
happens,  according  to  that  rule: 


^  http://www.5y5temdynamic5.orq/products/the-beer-qame/ 
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Figure  4.  Explosive  behavior  induced  by  the  direct  application  of  error  corrections  to  a  system  that  has  reversed  it 
input  states  by  the  time  the  correction  is  applied.  (Beer,  1979),  p.  60 


As  seen,  the  system  is  "exploding."  Why?  Because  the  signal  we  generate  to  correct  for  the  input  is  just 
one  phase  late,  so  instead  of  subtracting  when  it  sees  a  +1,  there  is  a  slight  delay  and  the  negative  signal 
we  generate  supplements  an  already  negative  signal,  making  it  even  more  negative. 


The  important  points  are:  this  is  common  and  is  the  result  of  the  structure  of  the  system,  both  static  and 
dynamic.  Our  methods  of  risk  identification  and  analysis  would  try  to  identify  such  connections  and 
delay. 
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B.2  RisKANDCoMPLExnY  Correlation 

Risk  can  be  defined  as  "a  measure  of  future  uncertainties  in  achieving  program  performance  goals  and 
objectives  within  defined  cost,  schedule  and  performance  constraints."{US  Department  of  Defense 
(Office  of  the  Undersecretary  of  Defense  (Acquisition,  2006  #1,  p.  1}. 


For  complex  defense  acquisition  programs,  often  various  types  of  risks  exist  that  manifest  themselves  at 
different  times  throughout  the  acquisition  process,  including  system  development.  These  risks  can  be 
technical,  programmatic  or  strategic  in  nature  and  can  result  in  substantial  cost  overruns,  delays, 
performance  issues,  reduced  adaptability  to  changing  requirements  or  even  total  cancellation  of  a 
project.  One  of  the  challenges  with  assessing  risk  using  the  traditional  risk  reporting  matrices  (See 
Figure  5)  for  complex  systems  acquisition  is  that  neither  the  likelihood  nor  the  true  consequence  of  a 
risk  can  be  objectively  established.  For  one,  there  is  substantial  uncertainty  around  the  interactions 
among  different  components  of  a  system  as  well  as  uncertainties  about  how  effectively  various  kinds  of 
risks  can  be  managed  across  a  multiplicity  of  interfaces. 


In  this  research  we  are  proposing  a  fundamentally  different  approach  to  risk  management,  one  that 
looks  at  how  complexities  within  the  technical  and  organizational  realms  result  in  uncertainties  that  can 
ultimately  lead  to  risks  in  the  system.  The  premise  of  this  research  is  that  in  the  realm  of  technical 
project  risk,  it  is  the  complexity  of  the  system  combined  with  the  experience/know-how  of  the 
contractors  that  determines  system  uncertainties  and  the  resulting  risks. 


1  2  3  4  5 


Consequence 

Figure  5.  Traditional  Risk  Reporting  Matrix  (US  Department  of  Defense  (Office  of  the  Undersecretary  of  Defense 
(Acquisition,  2006  #1} 
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The  objective  of  this  research  is  to  link  technical  complexity  with  uncertainty  and  risk  across  different 
stages  of  the  acquisition  process,  and  dynamically  quantifying  and  updating  risk  elements  for  decision¬ 
making  on  project  continuation,  modification  or  retirement. 

Complexity  may  be  the  root  cause  of  many  unforeseen  risks.  Program/project  complexity  per  se  can 
generate  negative  consequences  that  may  often  take  the  project  management  team  by  surprise. 
Common  and  advanced  methods  of  risk  modeling,  including,  for  example,  Bayesian  Networks,  cannot 
predict  the  sort  of  emerging  risks  that  manifest  continuous  ripple  effects  that  unfold  one  after  the  other 
almost  for  the  entire  duration  of  the  complex  projects.  This  type  of  intimidating  effect  of  complexity  is 
not  something  one  would  like  to  have  for  the  entire  program  duration,  or  perhaps  at  any  time  during 
the  program,  as  the  effect  is  one  of  being  out  of  control,  or,  indeed,  in  the  control  of  something 
unknown.  Often  the  complexity  manifests  in  risk  and  risk  creates  more  complexity.  This  is  known  as 
complexity-uncertainty  death  spiral.  In  several  case  studies  in  our  previous  research,  we  have  observed 
that  the  increase  in  structural  complexity  increases  the  risk  and  therefore  occurrence  of  the  minor 
undesired  event  (Efatmaneshnik  and  Nilchiani,  2012)(Nilchiani  and  Heydari,  2012).  The  unfolding  of  the 
first  risk  oftentimes  affects  the  structure  of  the  system  in  a  manner  that  increases  the  structural 
complexity.  The  incremental  increase  in  structural  complexity  again  can  contribute  to  the  next  risk  to 
unfold  and  the  spiral  escalation  can  continue.  We  model  this  process  by  hybrid  techniques  and  seek 
techniques  that  tackle  the  root  cause  of  hidden  risks  that  manifest  in  the  form  of  a  set  of  continuously 
mysterious  (no  clear  root  cause)  risks.  There  is  a  very  intricate  relationship  between  structural 
complexity  and  fragility  of  complex  Systems  of  Systems  that  can  be  the  result  of  an  escalation  of  overall 
system  sensitivity,  sometimes  in  a  very  short  time  period  (Figure  6). 


Figure  6.  The  Complexity-Risk  spiral.  Insignificant  uncertainties  and  risks  in  combination  with  structural  complexity 
escalate  into  a  fragile  situation  and  to  a  point  of  no  return  at  which  failure  is  certain. 
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Uncertainty  Modeling  and 
Correlation  Building: 

-Various  uncertainty  types  affect 
design  structure  matrix  differently 
-  Correlation  between  the  various 
uncertainty  types  and  the  order  of 
uncertain  events 


•  New  System’s  DSM  after  Uncertain  event  attl 

•  New  Complexity  Measure  after  tl 
•Modeling  the  possible  following  probable 
uncertainty  and  it's  effect  on  the  new  DSM  (Systems 
Dynamics  and  feedback  loops) 

•System  Failure  assessment 


0 


Figure  7.  Structural  complexity  and  risk  (uncertainty)  correlation  in  the  DARPA  F6  program. 


Figure  7  shows  an  example  of  the  structural  complexity  metrics  that  we  defined  and  used  for  the  DARPA 
F6  program  on  fractionated  space  systems  (Nilchiani,  2012).  Fractionated  space  systems  are  a  network 
of  satellites  in  orbit  that  can  consist  of  different  number  of  heterogeneous  satellites  with  various 
architectures  flying  in  formation.  Our  research  has  shown  a  direct  correlation  between  an  increase  in 
structural  complexity  and  how  fast  a  failure  or  risk  in  a  network  of  these  satellites  propagates  (such  as  a 
security  attack  on  one  of  the  satellites  in  the  network).  Figure  8  shows  some  of  the  results  of  the  F6 
simulation  that  connects  the  complexity  measure  of  the  system  to  the  mean  time  to  failure  for  various 
architectures  of  the  fractionated  space  systems. 
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Figure  8.  F6  Simulation  results  showing  that  increased  structural  complexity 
leads  to  shorter  time  to  failure  in  the  system. 


According  to  some  of  our  initial  studies  (Salado  and  Nilchiani,  2012;  Efatmaneshnik  and  Nilchiani,  2012), 
the  results  of  implementing  some  risk  mitigation  plans  can  create  ripple  effects  through  a  project  or 
system,  increase  the  complexity  of  the  system  and  therefore  lead  to  making  the  program  more 
vulnerable  to  known  risks  as  well  as  the  hidden  uncertainties.  Moreover,  the  existence  of  a  minimum  of 
only  three  interrelated  risks  with  significant  correlation  can  lead  to  a  ripple  effect  that  can  remain 
hidden  up  until  the  last  moment,  when  the  negative  consequences  become  fully  developed  and  surface, 
overwhelming  the  system.  Uncovering  these  types  of  hidden  cause  and  effect  relationships  requires 
thorough  structural  monitoring  of  the  system  requirements  and  design  as  early  as  possible  to  uncover  all 
the  dependencies  with  a  very  high  level  analysis. 


Additionally,  as  systems  demonstrate  more  functional  complexity,  they  can  perform  more  sophisticated 
missions.  However,  the  increased  functional  complexity  can  also  produce  an  increased  structural 
complexity  for  systems,  which  in  turn  increases  risks  of  failures.  While  more  complex  functionalities  are 
more  likely  to  deliver  higher  values,  structural  complexity  per  se  is  not  a  positive  attribute.  More 
complex  functions  can  require  that  structurally  complex  structures,  which  one  after  the  other  can  act  in 
unpredicted  ways.  In  essence,  functional  complexity  is  the  driving  force  behind  complexification.  Yet 
structural  complexity  is  a  cost  on  the  system,  because  it  increases  the  possibility  of  dramatic  response  to 
uncertainties,  or  fragility  (Figure  9). 
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Figure  9.  Logical  relationship  between  structural  and  functional  complexity 
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C.  Deriving  Project  Risk  from  Technical  Complexity  and  Contractor  Organizational 
Capabilities 

A  modified  risk  cube  that  looks  at  the  causal  relationships  among  technical  systems  complexity  and 
organizational  capability  in  dealing  with  technical  and  strategic  complexity  is  presented  in  the 
complexity-uncertainty-risk  environment  cube  of  Figure  10. 


Figure  10.  Complexity-Uncertainty-Risk  Environment  (CURE)  Cube 


Here  we  can  explore  the  interrelationships  among  various  aspects  of  technological  complexity  across 
the  acquisition  process  phases  and  explore  the  impact  of  organizational  complexity  (capability)  as  well 
as  strategic  (that  is,  higher  level,  as,  for  example,  at  the  mission  or  campaign  level)  complexity  on 
uncertainties  and  subsequently  risk.  The  unfolding  of  the  technical  complexity  depends  on  the  inherent 
requirements  complexity,  the  system  design,  and  the  capabilities  of  the  contractor  organization(s)  to 
match  their  internal  organizational  complexity  to  manage  the  technical  complexity.  In  our  current 
research  we  are  addressing  only  the  top  row  of  Figure  10,  the  technical  aspects  of  the  system  and  its 
creation  and  testing. 


Figure  11,  below,  illustrates  that  below  the  minimum  required  critical  complexity  a  system  cannot 
perform  the  functions  that  are  expected  and  above  a  maximum  tractable  complexity  level,  the  system 
development  process  can  spiral  out  of  control.  It  is  the  expertise,  know-how  and  experience  of  the 
contractor  organization,  working  with  the  government  acquisition  office  (where  both  use  standard 
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technical  management  processes,  such  as  version  control,  keeping  dependency  graphs  current,  keeping 
design  changes  in  harmony  with  requirements  changes,  etc.),  that  can  keep  the  development  process 
within  the  boundaries  of  these  two  and  stabilize  the  complexity  level  of  a  system.  Thus,  for  the  same 
system  but  different  contractor  +  acquisition  organizations,  the  graph  in  Figure  11  could  have  different 
forms. 

The  key  to  acquisition  risk  management  wiii  therefore  be  to  ensure  a  match  between  the  unfoiding 
technicai  complexity  with  the  internal  organization,  know-how  and  expertise  of  the  contractor(s)  + 
acquirer  in  managing  complexity. 


Complexity 


Concept  Program  Technology  Production  and 

Exploration  Definition  Development  Fielding 


Figure  11.  Complexity  evolution  throughout  the  systems  acquisition  lifecycle 


D.  Architectural-level  Complexity  Measures  for  Acquisition  Systems: 

Summary  of  Case  Studies 

The  first  step  therefore  in  transitioning  towards  a  complexity-centric  risk  assessment  is  to  be  able  to 
measure  systems  complexity  over  the  acquisition  process.  As  it  is  possible  that  there  is  no  detailed 
design  in  the  early  stages  of  the  acquisition  process,  the  measurement  of  complexity  has  to  start  at  the 
architectural  (high-level  requirements)  level.  Tables  1  and  2  summarize  the  different  types  and  planes 
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of  acquisition  complexity  at  the  architectural  level.  It  should  be  noted  that  the  following  Tables 
summarize  some  of  the  major  variables  that  contribute  to  the  increased  complexity  of  the  system. 
However  the  list  and  variables  may  not  be  comprehensive  and  in  phase  2  of  the  project,  we  are  aiming 
at  identifying  the  majority  of  variables  that  contribute  to  the  complexity  of  the  system 


Table  1.  Six  types  of  complexity  (Source:  Sheard,  2012) 


Six  Types 

Structural:  Size 

Number  of  elements,  number  of  instances,  total  cost,  total  number  of 
requirements 

Structural:  Connectivity 

Number  of  connections,  density  of  connections,  strengths  of 
relationships,  amount  of  conflict,  distribution  of  connectedness 

Structural:  Inhomogeneity 

Number  of  different  types  of  entities,  number  of  types  of  relationships, 
number  of  different  areas  within  a  space,  diversity  of  sizes  of  elements  or 
contractors  or  stakeholders 

Dynamic:  Short-term 

Existence  of  loops/circuits,  safety-criticality,  tendency  to  blow  up  in 
operational  time  frame,  seriousness  of  consequences  of  a  mishap 

Dynamic:  Long-term 

Evolution  of  purpose  of  an  item,  co-evolution  of  a  variant  and  its 
environment,  how  much  different  the  next  iteration  of  a  system  might  be 

Socio-political 

Fraction  of  stakeholder  interests  that  are  based  on  power,  amount  of 
disagreement  among  stakeholders,  number  of  layers  of  management, 
changes  of  opinion  of  management  or  stakeholders,  number  of  different 
cultures  working  together  on  a  project,  inhomogeneity  of  stakeholder 
utilities. 
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Table  2.  Four  planes  of  acquisition  complexity  (Source:  Sheard,  2012) 


Four  Entities 

[Technical]  System  being  built 

Product,  system,  system-of-systems,  tank,  squadron, 
database,  sensor,  software  algorithm. 

Project  or  organization  doing  the 
building 

Project,  organization,  program,  tasks,  team 

Environment,  both  external  systems  and 
people 

Customers,  buyers,  market,  external  technological  system, 
future  systems  that  need  to  interface  with  product 

Cognitive:  capacity  of  humans  to 
understand,  conceive  of,  build  and 
operate  the  system. 

Learning  curve,  uncertainty,  confusion,  operator  skill  set. 
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E.  Measuring  Architectural  Level  Complexity:  Initial  Explorations 

Based  on  a  comprehensive  literature  and  state  of  the  art  review  we  have  converged  on  the  following 
five  lenses  for  measuring  complexity.  Should  these  prove  to  be  inadequate  for  our  research,  we  will 
devise  new  ones  based  on  our  own  observations  of  systems.  We  will  explore  which  of  the  following 
lenses  of  complexity  measurements  applied  to  an  architecture-level  systems  description  can  dynamically 
predict  acquisition  risk  and  improve  mid-process  decision-support 

1.  Requirement  critical  (algorithmic)  complexity 

2.  Critical  control  path  (cyclomatic)  complexity 

3.  Dynamic  architectural  complexity 

4.  Structural  complexity 

5.  Modified  architectural-structural  complexity 

We  will  then  explore  how  the  experience  of  contractors  +  government  plays  a  role  in  managing  the 
complexity  of  the  system  in  the  acquisition  process. 


1)  Requirement  Critical  Complexity 

Requirement  critical  complexity  refers  to  the  minimum  amount  of  complexity  a  system  needs  to  have  in 
order  to  perform  a  desired  set  of  functions  in  line  with  expressed  requirements.  Based  on  Kolmogorov 
complexity  metric,  it  refers  to  the  minimum  set  of  architectural  level  components  and  linkages  {y}  that 
would  address  requirement  set  {x}.  In  other  words,  the  requirement  critical  complexity  of  a  system  can 
be  expressed  as  the  minimal  systems  architecture  {y}  (minimum  number  and  type  of  components  and 
linkages)  that  would  theoretically  produce  performance  set  {x}. 


The  determination  of  {y}  given  {x}  is  an  important  research  question  and  this  research  will  try  to 
establish  this  threshold  for  various  kinds  of  complex  systems.  The  calculation  of  requirement  critical 
complexity  can  be  done  either  through  modified  structural  complexity  metrics  that  will  be  discussed 
further  in  this  document. 


2)  Critical  Control  Path  Complexity 

Based  on  the  concept  of  cycolmatic  complexity  in  software,  the  critical  control  path  complexity  metric 
measures  the  number  of  linearly  independent  control  paths  through  a  systems  architecture  graph.  This 
number  changes  as  the  architecture  (or  the  resulting  design)  changes  over  time  and  is  estimated  by  the 
following  equation: 


CCP{t)  =  ^  n{t,  i)  —  Z(t,  i)  +  p{t,  i) 
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where  CCP{t)  is  the  critical  control  path  complexity  at  time  t,  n(t,i)  is  the  number  of  nodes  in  the 
connected  graph  of  the  architecture  expression  for  module  (i),  /  is  the  number  of  linkages  at  time  t  in 
module  (i)  and  p  is  the  number  of  distinct  connected  components  in  the  architectural  flow  graph.  And 
the  sum  is  over  all  modules. 


3)  UML-based  Five-Views  Dynamic  Architectural  Complexity  (Lankford) 

Rather  than  a  single  number,  the  UML-based  Five  Views  Dynamic  Architectural  complexity  metric  allows 
the  measurement  of  various  system  complexity  metrics  over  time. 

The  five  views  complexity  vector  is  calculated  as  follows: 


C5l^(t)  = 


^class(,^>  0 
^process  (t,  0 
^component  (pi  0 
^interfacesO'i  0 
^patternsO'i  0 


Where: 


Within 

Cciass(f>  0=Number  of  classes  and  objects  within  each  module  at  time  t 
Cprocess(P'  0  =Number  of  processes  and  threads  within  each  module  at  time  t 
(^component(t>  0  =Number  of  components  =  the  number  of  nodes 
C inter faceiP,  0  =Number  of  interfaces  between  each  of  these 
Cpattern  (f-  0  =Number  of  identifiable  design  patterns  within  each  module 


4)  Simple  Structural  Complexity  (Meyer) 

Simple  structural  complexity  can  provide  an  easy  to  calculate  way  to  capture  how  the  complexity  of  a 
system  is  changing,  by  calculating  changes  in  the  number  of  parts  (or  sub-systems  or  systems),  types  of 
parts  and  number  of  interfaces  over  time. 


^structurali.^f  0  “  ^  0  ^  0 

Where 

Np{t,  i)  =Number  of  parts/subsystems  in  subsystem/system  ;  at  time  t 
Ny(t,  0=  Types  of  parts/subsystems  in  subsystem/system  /  at  time  t 
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Nx{t,  i)=  Number  of  interfaces  in  subsystem/system  ;  at  time  t 


5)  Modified  Architectural-Structural  Complexity  (MASC) 

The  modified  architectural-structural  complexity  is  the  most  comprehensive  measure  of  architectural 
complexity,  taking  into  account  size,  type,  interconnections  and  interfacial  complexity  of  architectural 
modules  into  consideration.  It  is  based  on  Kinnunen  (2000).  Modifying  the  simple  architectural 
complexity  equation  for  MASC  we  get: 


CmasM  =  ^V[(iVp(t)  X  Nyit,  0]  X  X  Nabjit)  X  Napit)]  X 

Where  the  arguments  are  respectively: 

•  Number  of  distinct  types  of  objects/components 

•  Number  of  objects  within  each  type 

•  Number  of  processes/functionalities  affecting  an  object 

•  Number  of  objects/components  affecting  a  process 

•  Number  of  operations  per  process 

•  Number  of  interfaces  weighted  with  the  interface  complexity  multiplier  (ICM)  (related 
to  the  integration  readiness  levels  (IRLs)  between  different  systems/subsystems). 

It  should  be  noted  that  these  five  types  of  architectural  level  complexity  measures  are  our  initial 
exploration  of  the  relevant  complexity  measures  of  the  technical  system.  Our  research  team  may  have 
to  define  novel  measures  based  on  the  existing  literature  on  complexity  that  may  be  more  useful  for 
different  milestones  of  an  acquisition  program,  and  in  particular  characterizing  the  dynamic  behavior  of 
the  architecture. 


C.  The  Fit  Between  Technical  Risk  of  the  Product  and  an  Organization's  Capability  to  Manage  it 


What  accounts  for  one  enterprise  being  able  to  create  a  complex  product  and  another  not?  The  primary 
conjecture,  attributed  to  Ashby  (Ashby,  1961),  is  that  the  successful  enterprise  that  can  construct  a 
complex  product  has  enough  "variety"  (he  called  it  requisite  variety)  in  the  way  it  is  organized  and 
applies  its  resources.  Variety  is  diversity,  ability  to  react  to  various  problems  and  opportunities,  including 
unexpected  ones. 


Perhaps  one  of  the  most  vivid  illustrations  of  variety  in  this  context  was  during  the  Apollo  13  manned 
space  flight  in  1970,  when  an  oxygen  tank  aboard  exploded,  limiting  power,  causing  loss  of  cabin  heat, 
reducing  the  availability  of  potable  water,  and  increasing  the  concentration  of  carbon  dioxide  in  the 
cabin  air.  It  was  the  mounting  concentration  of  carbon  dioxide  that  proved  most  troubling,  as  the 
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astronauts  would  die  of  lack  of  oxygen  if  it  were  not  reversed.  A  team  on  the  ground  was  assembled  and 
given  the  task  of  figuring  out  how  to  create  a  carbon  dioxide  removal  system,  given  the  constraints  on¬ 
board.  That  the  ground  team  succeeded  was  a  tribute  to  its  variety,  its  diversity  of  thought,  as  it  quickly 
suggested  and  tested  numerous  options. 


One  of  the  biggest  challenges  in  using  variety  to  characterize  organizations  is  that  it  is  so  difficult  to 
observe,  to  measure.  Two  authors  (Beer,  1979;  Jaques,  2006)  have  suggested  antidotes  to  this  and  we 
are  exploring  their  methods. 


D.  The  Places  of  Case  Studies  and  Quantitative  Data 

We  are  seeking  to  know  what  programs  and  organizations  are  "made  of"  that  might  inform  the 
identification  of  risk.  Our  premise  is  that  complexity  is  a  major  indicator  of  risk.  In  order  to  validate  or 
invalidate  the  premise  we  need  data.  The  most  convincing  data  would  be  numeric,  quantitative  that 
showed  the  relationship  between  product  complexity,  say,  and  risk.  If  that  data  are  not  available,  then 
we  might  use  case  studies. 


Since  we  do  not  yet  have  quantitative  data,  we  have  indeed  been  reading  case  studies,  supplied  to  us  by 
the  deep  reservoir  provided  by  our  colleague.  Dr.  Gary  Witus,  at  Wayne  State  University.  At  one  stage 
he  supplied  15  cases,  some  with  multiple  artifacts.  Dr.  Mostashari  read  them  and  in  the  end  was  not 
able  to  deduce  anything  general. 


Dr.  Witus  responded  to  our  request  for  additional  case  studies  and  we  have  not  yet  had  a  chance  to 
absorb  them,  and  it  is  a  priority  for  our  next  steps. 


At  some  point  -  earlier  is  better  -  we  are  going  to  need  access  to  quantitative  data  that  will  help  us 
confirm  or  deny  the  connection  between  some  measure  of  complexity  and  technical  program  risk.  This, 
too,  is  a  priority  for  the  next  steps. 


Both  case  studies  and  access  to  quantitative  program  information  will  help  us  steer  where  to  look 
deeper,  help  us  consider  what  programs  are  made  of.  In  the  end,  it  is  possible  that  programs  do  not 
collect  the  measures  of  complexity  that  we  think  are  the  most  indicative  of  risk,  so  we  will  have  to  work 
with  programs  on  a  pilot  basis  to  install  new  measures  and  assess  the  ability  of  those  measures  to 
predict  technical  risk. 
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Examples  and  Some  Case  Studies 


Concept  Demonstration:  Complexity  and  Risk 

One  of  the  easy  ways  to  characterize  complexity  for  a  system  at  an  architectural  level  is  to  analyze  the 
number  of  different  interactions  between  the  different  subsystems. 

T  “  1) 

interactiom^^^  = - 


Interactions  can  be  spatial,  material,  energy  and/or  information.  If  we  look  at  aircraft  designs  over  the 
years  we  can  see  how  avionics  has  become  more  complex. 


Year  of  Initial  Operational  Capability 


Figure  12.  Increasing  avionics  complexity  (dimensionless)  over  the  years  (Source:  Diterick,  2010) 

Analyzing  the  relationship  between  cost  and  development  schedule  in  154  DoD  projects,  McNutt  (2000) 
estimates  the  relationship  between  cost  and  complexity  to  be  estimated  by  the  following  equation: 

De\'eJopmentCost  =  {Q.Q?>y~De\'elopmentTime^^^^^^^^  +1.36)^ 

Where  the  development  cost  is  in  millions  of  USD. 
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Case  Studies 

An  analysis  of  the  following  31  acquisition  programs  with  a  calibrated  complexity  metric  shows  the 
following  correlation: 


Figure  13.  Cost  overrun  as  a  function  of  log  calibrated  architectural  systems  complexity 
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Figure  14.  Cost  overrun  and  schedule  slips  for  different  types  of  weapons  systems.  Most  cost  overruns  occur  for 
ship  systems,  while  most  schedule  slips  happen  for  aircraft.  Avionic  systems  have  had  a  good  track  record  of 
beating  both  cost  and  schedule  plans. 
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C-130 

E2-D  Advanced 
Hawkeye 
F-35 
FAB-T 

Global  Hawk 
Grey  Eagle 
HC-130 
MQ-4C  UAV 
P-8A  Poseidon 
Reaper UAV 
Excallbur  Guided 
Artillery 
IDECOM 
Joint  Precision- 
Apparoach  and 
Landing  System 
Airbrone  and 
Tactical  Radio 
System 
Joint  Tactical 
Radio  System 
Handheld 


$17,747 

$326,535 

$4,688 

$12,812 

$5,159 

$13,091 

$13,052 

$32,969 

$11,919 

$1,781 

$821 


$26,575 


$8,160 


$8,358 


Aircraft 


Aircraft 

Aircraft 

Aircraft 

Aircraft 

Aircraft 

Aircraft 

Aircraft 

Aircraft 

Aircraft 

Artillery 
Avionic  System 


Avionic  System 


Boeing 


Northrup  Gruman 
Lockheed  Martin 
Boeing 

Northrup  Gruman 
General  Atomics 
Lockheed  Martin 
Northrup  Gruman 
Boeing 

General  Atomics 

Raytheon 
ITT  Electronics 


Raytheon 


Jiedule  Type  of 
jlip  I^AcquisIlJ^ 

0  High  TRL 


Communication  System  Lockheed  Martin 


Communication  System  General  Dynamics 


20.3 

78.2 
29.1 

172.2 
-18 
-5.1 
1.6 
0.1 
18.9 

282.4 

-0.5 


-2.9 


0.1 


43.2 
N/A 

35 

127.3 

N/A 

N/A 

0 

0 

19 

27.2 
-8.5 


2.7 


13.8 


22.4 


Medium  TRL 
Low  TRL 
Medium  TRL 
Low  TRL 
High  TRL 
High  TRL 
High  TRL 
High  TRL 
Medium  TRL 

Medium  TRL 
High  TRL 


High  TRL 


Medium  TRL 


Medium  TRL 


Mobile  User 
Objective  System 
Navy  Multi-band 
Terminal 
Warfighter 
Information 
Network  Tactical 
Apache  block  IIIA 
CH-53 
AGM88E 
Army  Integrated 
Air  and  Missile 
Defense 

Joint  Land  Attack 
Cruise  Missile 
Defense 
Standard  Missile 
RAM 
CVN  78 
DDG  1000 
Joint  Highspeed 
Vessel 


$6,978 

$1,214 


$6,052 

$10,737 

$22,439 

$1,902 


$5,529 


$7,858 

$6,297 

$33,994 

$20,985 

$3,674 


Communication  System  Lockheed  Martin  3.8 
Communication  System  Raytheon  -11.2 


Communication  System  General  Dynamics  8.6 
Helicopter  Boeing  39.7 

Helicopter  Sikorsky  5.7 

Missile  ATK  Missile  Systems  10.9 


Missile 


Missile 

Missile 

Ship 

Ship 

Ship 


Northrup  Gruman 


Raytheon 

Raytheon 
Huntington  Ingalls 
BAE  Systems 

Austral  USA 


9.9 


18 

10.5 

-4.4 

543 


28.9  Medium  TRL 

0  High  TRL 


42 

3.8 

32 

22.4 


1.3 

6.2 

25.3 
13.1 

73 

4.2 


Medium  TRL 
High  TRL 
High  TRL 
High  TRL 


High  TRL 


Medium  TRL 

Medium  TRL 
High  TRL 
Low  TRL 

High  TRL 


LHA  Replacement 
Assault  Ship 
LCS 
GPS  III 

Space-Based  IR 
System  (SBIRS) 


$10,096 

$32,867 

$4,210 

$18,266 


Ship 

Ship 

Space  System 
Space  System 


Huntington  Ingalls  5.8 

Lockheed  Martin  76 

Lockheed  Martin  6.8 


Lockheed  Martin 


231.2 


13 

183 

N/A 

N/A 


High  TRL 
Low  TRL 
Medium  TRL 

Low  TRL 


Table  3.  A  selection  of  case  studies  of  DoD  acquisition  program  and  their  percentage  of  cost  and  schedule 

overruns. 
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Average  Program  Size 
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■  Average  Program  Size 


Figure  IS.Average  program  size  (total  program  cost)  of  case  studies  explored  in  this  research  (in  million  $) 


Additional  Case  Studies  (Not  included  in  Complexity  Analysis) 

The  following  are  additional  case  studies  the  team  looked  at,  but  most  were  suffering  from  program 
complexity  rather  than  technical  complexity. 


A-lOlhUNDERBOLTlI  (AIRCRAFT) 

Acquisition  Organization:  U.S.  Air  Force 

Risks  and  Weaknesses 

■  Technical:  Concurrent  development  of  a  new  technology  (the  GAU-8/A  gun  system)  and  the 
aircraft  at  the  same  time  with  the  aircraft  architecture  revolving  around  the  armament  system 
created  delays  in  the  acquisition  process.  Also  the  original  structural  design  proved  inadequate 
for  the  design  life,  and  even  fixes  during  production  were  inadequate  for  all  but  the  latest 
aircraft  produced. 

■  Programmatic:  Overlooked  problems  associated  with  production  readiness  and  contractor 
financial  stability  did  not  go  away  and  had  to  be  resolved  far  too  late  in  the  development 
program.  Additional  problems  included  loss  of  the  Original  Equipment  Manufacturer  (OEM),  on- 
again/off-again  decisions  to  retire  the  A-10,  unstable  funding  for  inspection  and  repair,  and 
major  personnel  disruptions  resulting  from  a  BRAC  decision.  Critical  "health  of  the  fleet" 
structural  inspections  were  not  performed  during  sustainment,  and  a  subsequent  repair 
program  failed  to  provide  the  desired  level  of  life  extension. 

Strengths 

Close  attention  to  key  mission  characteristics  (lethality,  survivability,  responsiveness,  and 
simplicity)  allowed  the  concept  formulation  and  subsequent  system  design  to  result  in  an 
effective  CAS  aircraft,  and  design-to-cost  goals  kept  the  government  and  contractor  focused  on 
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meeting  the  critical  requirements  at  an  affordable  cost.  The  A-10  did  not  meet  all  its  cost  goals, 
but  it  came  much  closer  to  them  than  most  major  defense  development  programs  did  in  that 
time  frame  or  since  then. 

Complexity  Factors  Leading  to  Risk 

•  Low  TRL  technology  at  core  of  systems  architecture  (Interface  complexity) 

•  Requirement  changes  rendering  architecture  inadequate  (Requirement  Complexity) 

Contractor  technical  and  financial  capability  (Organizational  Requisite  Complexity) 

Source:  A-10  Thunderbolt  II  (Warthog)  SYSTEMS  ENGINEERING  CASE  STUDY ,  Air  Force  Center  for 

Systems  Engineering 


C  -  5A  Galaxy  (Airc  RAFi) 

Acquisition  Organization:  U.S.  Air  Force 

Risks  and  Weaknesses 

■  Technical:  A  Weight  Empty  Guarantee  was  included  in  the  specification  as  a  performance 
requirement  and  in  the  contract  as  a  cost  penalty  for  overweight  conditions  of  delivered  aircraft. 
The  aircraft  Weight  Empty  Guarantee  dominated  the  traditional  aircraft  performance 
requirements  (range,  payload,  etc.),  increased  costs,  and  resulted  in  a  major  shortfall  in  the  wing 
and  pylon  fatigue  life.  The  stipulation  of  a  Weight  Empty  Guarantee  as  a  performance 
requirement  had  far-reaching  and  significantly  deleterious  unintended  consequences. 

■  Programmatic:  The  Total  Package  Procurement  Concept  (TPPC)  employed  by  the  government 
required  a  fixed-price,  incentive  fee  contract  for  the  design,  development,  and  production  of  58 
aircraft.  It  included  a  clause  giving  Total  Systems  Performance  Responsibility  (TSPR)  to  the  prime 
contractor.  TPPC  was  invented  to  control  costs,  but  it  was  the  underlying  cause  of  the  cost 
overrun  and  limited  the  number  of  aircraft  purchased  under  the  original  contract 

Strengths 

The  process  for  developing  and  documenting  the  system  performance  requirements  involved 
the  User  (warfighter),  planners,  developers,  and  technologists  from  both  the  government  and 
industry  in  a  coordinated  set  of  trade  studies.  It  resulted  in  a  well-balanced,  well-understood  set 
of  requirements  that  fundamentally  remained  unchanged  throughout  the  program. 

Complexity  Factors  Leading  to  Risk 

•  Detrimental  hard  requirement  with  cascading  effect  on  mission  critical  requirements  and 
architectural  design  (Requirement  complexity) 

•  Weight,  wing  and  pylon  design  conflict  (Interfacial  complexity) 

•  Faulty  procurement  concept  (Organizational  Process  Complexity) 

Source:  C-5A  Galaxy  Systems  Engineering  Case  Study,  Air  Force  Center  for  Systems  Engineering 


Contract  Number:  H98230-08-D-0171  TO  0030,  RT040 

Report: No.  SERC-2013-1R-040-3  Volume  I 
November  12,  2013 

37 


UNCLASSIFIED 


F- 111  (Aircraft) 

Acquisition  Organization:  U.S.  Air  Force  and  U.S.  Navy 

Risks  and  Weaknesses 

■  Technical:  The  F-111  acquisition  process  suffered  from  a  nearly  impossible  multi-role/multi- 
service  requirement  specification,  and  a  protracted  development  cycle  in  which  numerous 
serious  technical  problems  had  to  be  identified  and  corrected.  Of  the  1,726  total  aircraft  buy 
that  had  originally  been  planned  in  1962,  only  562  production  models  of  seven  different  variants 
were  completed  when  production  ended  in  1976.  The  F-111,  like  any  complex  weapon  system 
development  program,  which  provides  new  war-fighting  capability,  had  areas  of  risk  or 
deficiency  that  came  to  light  during  RDT&E  even  though  there  was  perceived  low  risk  in  the 
design.  The  F-111  development  program  introduced  concurrency  (overlap)  between  design 
validation/verification  and  production  to  accelerate  program 

■  Programmatic:  Systems  Architecture  and  Design  Trade-Offs  were  not  performed  to  achieve  an  F- 
111  design  that  was  balanced  for  performance,  cost  and  mission  effectiveness  (including 
survivability)  and  the  attendant  risk  and  schedule  impacts.  The  F-111  suffered  from  poor 
communications  between  the  Air  Force  and  Navy  technical  staffs,  and  from  over-management 
by  the  Secretary  of  Defense  and  the  Director,  Defense  Research  and  Engineering,  and  it  came 
under  intense  congressional  scrutiny,  which  restricted  the  System  Program  Office  (SPO)  Director 
from  applying  sound  systems  engineering  principles. 

Complexity  Factors  Leading  to  Risk 

•  Impossible  requirements  with  severe  conflicts  (Requirement  complexity) 

•  Inadequate  verification  and  validation  (Organizational  Process  Complexity) 

•  Multi-agency  acquisition  process  (Organizational  Process  Complexity) 

•  Sociopolitical  sensitivity  (Organizational  Process  Complexity) 

Source:  Fill  Systems  Engineering  Case  Study,  Air  Force  Center  for  Systems  Engineering 
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AGM-88E  Advanced  Anti-Radiation  Guided  Missile  (AARGM) 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  ATK  Missile  Systems 

As  of 

Latest 

Percent 

Company 

07/2003 

06/2011 

change 

Program  office:  Patuxent  River,  MD 

Research  and  development  cost 

$637.2 

$722.2 

13.4 

Funding  needed  to  complete; 

Procurement  cost 

$963.6 

$1,180.0 

22.5 

R&D:  $0.0  million 

Total  program  cost 

$1,600.7 

$1,902.3 

18.8 

Procurement:  $1,319.6  million 

Program  unit  cost 

$.894 

$-991 

10.9 

Total  funding;  $1,319.6  million 

Total  quantities 

1,790 

1,919 

7.2 

Procurement  quantity:  1 ,767 

Acquisition  cycle  time  (months) 

85 

104 

22.4 

Apache  Block  IIIA 


Program  Essentials 
Prime  contractor  Boeing 
Program  office:  Huntsville,  AL 
Funding  needed  to  complete: 
R&D:  S706.8  million 
Procurement:  $8,363.7  million 
Total  funding;  $9,070.6  million 
Procurement  quantity;  610 


Program  Performance  (fiscal  year  2012  dollars  in  millions) 


Research  arxJ  development  cost 

As  of 
08/2006 
$1,155.6 

Latest 

12/2010 

$1,640.3 

Percent 

change 

41.9 

Procurement  cost 

$6,086.9 

$9,096.8 

49.4 

Total  program  cost 

$7,242.5 

$10,737.0 

48.3 

Program  unit  cost 

$12,031 

$16,803 

39.7 

Total  quantities 

602 

639 

6.1 

Acquisition  cycle  time  (months) 

79 

82 

3.8 

The  latest  cost  and  quansoes  do  not  nctuoe  tne  E7  neie-ouiid  nelcopters  atat  are  oeing  acquireo 
under  tne  AB3B  major  de'ense  acquismon  program. 


Army  Integrated  Air  and  Missile  Defense 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor.  Northrop  Grumman 

As  of 

Latest 

Percent 

Space  &  Mission  Systems  Corp. 

12/2009 

08.^2011 

change 

Program  office:  Huntsville.  AL 

Research  and  developrrtent  cost 

$1,595.2 

$2,019.8 

26.6 

Funding  needed  to  complete: 

Procurement  cost 

$3,433.4 

$3,509.0 

2.2 

R&D:  $1 ,370.5  million 

Total  program  cost 

$5,028.6 

$5,528.8 

9.9 

Procurement:  $3,509.0  million 

Program  unit  cost 

$16,988 

$18,678 

9.9 

Total  funding;  $4,879.5  million 

Total  quantities 

296 

296 

0.0 

Procurement  quantity:  285 

Acquisition  cycle  lime  (months) 

80 

81 

1.3 
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C-130  Avionics  Modernization  Program 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  m 

illions) 

Prime  contractor  Boeing 

As  of 

Latest 

Percent 

Program  office:  Wright-Patterson  AF8, 

07/2001 

12/2010 

change 

OH 

Research  and  development  cost 

$775.4 

$1,948.3 

151.3 

Funding  needed  to  complete: 

Procurement  cost 

$3,356.8 

$4,256.0 

26.8 

Ft&D:  $42.2  million 

Total  program  cost 

$4,132.3 

$6,204.3 

50.1 

Procurement:  $3,890.4  million 

Program  unit  cost 

$7,962 

$28,074 

252.6 

Total  fundirrg;  $3,932.6  million 

Total  quantities 

519 

221 

-57.4 

Procurement  quantity:  208 

Acquisition  cycle  time  (months) 

NA 

NA 

NA 

CH  53-K  Heavy  Lift  Replacement 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

F>rime  contractor  Sikorsky  Aircraft 

As  of 

Latest 

Percent 

Corporation 

12/2005 

08./2011 

change 

Program  office:  Patuxent  River.  MD 

Research  and  development  cost 

$4,378.9 

$6,058.1 

38.3 

Funding  needed  to  complete: 

F>rocurennent  cost 

$12,178.3 

$16,381.7 

34.5 

R&D:  $3,252.9  million 

Total  program  cost 

$16,557.1 

$22,439.9 

35.5 

Procurement:  $16,381.7  million 

Program  unit  cost 

$106,136 

$112,199 

5.7 

Total  funding:  $19,634.6  million 

Total  quantities 

156 

200 

28.2 

Procurement  quantity:  196 

Acquisition  cycle  time  (months) 

119 

157 

31.9 

CVN  78  Class 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  Huntington  Ingalls 

As  of  Latest 

Percent 

Industries-Newport  News 

04/2004  08/2011 

change 

Program  office:  Washington.  DC 

Research  and  development  cost 

$4,803.3  $4,646.8 

-3.3 

Funding  needed  to  complete; 

Procurement  cost 

$30,770.8  $29,346.8 

^.6 

R&D:  $827.7  million 

Total  program  cost 

$35,574.1  $33,993.6 

-4.4 

Procurement:  $16,540.9  million 

Program  unit  cost 

$11,858,040  $11,331,185 

■4  A 

Total  funding:  $17,368.6  million 

Total  quantities 

3  3 

0.0 

Procurement  quantity:  2 

Acquisition  cycle  time  (months) 

137  155 

13.1 
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DDG  1000  Destroyer 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  BAE  Systems,  Bath 

As  of 

Latest 

Percent 

Iron  Works,  Huntington  Ingalls 

01/1998 

08/2011 

change 

Industries,  Raytheon 

Research  and  development  cost 

$2,277.9 

$10,378.4 

355.6 

Program  office;  Washington,  DC 

FVocurement  cost 

$32,522.1 

$10,607.2 

.67.4 

Funding  needed  to  complete: 

Total  program  cost 

$34,800.0 

$20,985.6 

-39.7 

R&D:  $995.6  million 

Program  unit  cost 

$1,087,500 

$6,995,214 

543.2 

Procurement:  $1,418.1  million 

Total  quantities 

32 

3 

-90.6 

Total  funding;  $2,413.6  million 

Acquisition  cycle  time  (months) 

128 

222 

73.4 

Procurement  quantity:  0 

E2-D  Advanced  Hawkeye 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  Northrop  Grumman 

As  of 

Latest 

Percent 

Program  office:  Patuxent  River.  MD 

06/2003 

08.'2011 

change 

Funding  needed  to  complete; 

Research  and  development  cost 

$3,841.0 

$4,537.9 

18.1 

R&D: *$367.8  million 

Procurement  cost 

$10,911.1 

$13,167.1 

20.7 

Procurement:  $10,874.7  million 

Total  program  cost 

$14,752.0 

$17,747.3 

20.3 

Total  funding:  $11,257.5  million 

Program  unit  cost 

$196,694 

$236,630 

20.3 

Procurement  quantity:  60 

Total  quantities 

75 

75 

0.0 

Acquisition  cycle  time  (months) 

95 

136 

43.2 

Excalibur  Precision  Guided  Artillery 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  m 

illions) 

Prime  contractor  Raytheon 

As  of 

Latest 

Percent 

Program  office:  Picatinny  Arsenal,  NJ 

02/2003 

08/2011 

change 

Funding  needed  to  complete: 

Research  and  development  cost 

$765.5 

$1,068.3 

39.6 

R&D:  $50.2  million 

Procurement  cost 

$4,010.8 

$712.1 

-82.2 

Procurement:  $234.9  million 

Total  program  cost 

$4,776.2 

$1,780.5 

-62.7 

Total  furtding:  $285.1  million 

Program  unit  cost 

$.062 

$.238 

282.4 

Procurement  quantity:  3,455 

Total  quantities 

76,677 

7.474 

-90.3 

Acquisition  cycle  time  (months) 

136 

173 

27.2 

Ibtal  quantities  inctuae  3.as£  mcrenent  ID  pmjeclies 
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F-35  Lightning  II 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

FHime  contractor.  Lockheed  Martin, 

As  of 

Latest 

Percent 

Pratt  and  Whitney 

10/2001 

12/2010 

change 

Program  office:  Arlington,  VA 

Research  and  development  cost 

$38,976.7 

$58,387.6 

49.8 

Funding  needed  to  complete: 

Procurement  cost 

$172,921,4 

$267,595.6 

54.7 

R&D:  510,117.8  million 

Total  program  cost 

$213,708.2 

$326,535.2 

52.8 

Procurement:  $245,676.5  million 

Program  unit  cost 

$74,567 

$132,900 

78.2 

Total  funding:  $255,970.4  million 

Total  quantities 

2,866 

2.457 

-14.3 

Procurement  quantity:  2,353 

Acquisiton  cycle  time  (months) 

116 

TBD 

NA 

.atest  column  does  not  ruiiy  re'^ect  tne  resaucturea  JSF  program. 

Costs  are  expected  to  grow  and  me 

schedule  vrti  De  ertended 

Family  of  Advanced  Beyond  Line  of  Sight  Terminals  (FAB-T) 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  Boeing 

As  of 

Latest 

Percent 

Program  office:  Hanscom  AFB,  MA 

12/2006 

08/2011 

change 

Funding  needed  to  complete: 

Research  and  development  cost 

$1,537.1 

$2,338.7 

52.2 

R&D:  $571 .4  million 

Procurement  cost 

$1,651.4 

$2,349.6 

42.3 

Procurement:  $2,338.6  million 

Total  program  cost 

$3,188.5 

$4,688.3 

47.0 

Total  funding;  $2,910.0  million 

F>rogram  unit  cost 

$14,762 

$19,058 

29.1 

Procurement  quantity:  216 

Total  quantities 

216 

246 

13.9 

Acquisition  cycle  time  (months) 

129 

174 

34.9 

Tne  ates;  cost  data  do  nor  reflect  me  current  cos:  or  tne  program.  A  new  acguKioon  program  oaseiirve 

nas  rxx  yet  Deen  approved. 

Global  Hawk 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  Northrop  Gmmman 

As  of 

Latest 

Percent 

program  office:  Wright-Patterson  AFB, 

03/2001 

10/2011 

change 

OH 

Research  and  development  cost 

$1,041.6 

$4,769.3 

357.9 

Funding  needed  to  complete: 

F*rocurement  cost 

$4,318.8 

$7,877.4 

82.4 

R&D:  $1,657.1  million 

Total  program  cost 

$5,392.0 

$12,811.6 

137.6 

Procurement:  $3,098.9  million 

Program  unit  cost 

$85,588 

$232,938 

172.2 

Total  funding;  $4,789.4  million 

Total  quantities 

63 

55 

-12.7 

Procurement  quantity:  13 

Acquisition  cycle  time  (months) 

55 

125 

127.3 

Contract  Number:  H98230-08-D-0171 
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Global  Positioning  System  III 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Pritne  contractor  Lockheed  Martin 

As  of 

Latest 

Percent 

Program  office:  B  Segundo.  CA 

05/2008 

08/2011 

change 

Funding  needed  to  complete; 

Research  and  development  cost 

$2,524.2 

$2,694.8 

6.8 

R&D:  $924.6  milhon 

Procurement  cost 

$1,417.2 

$1,515.8 

7.0 

Procurement:  $1,435.0  million 

Total  program  cost 

$3,941.4 

K210.6 

6.8 

Total  funding:  $2,359.6  million 

F>rogram  unit  cost 

$492,672 

$526,323 

6.8 

Procurement  quantity;  6 

Total  quantities 

8 

8 

0.0 

Acquisition  cycle  time  (months) 

NA 

NA 

NA 

We  cou  Id  not  calculate  acquiEitior  cycle  Dines  tor  me  tirs;  mcrenen;  or  me  GPS  III  program  oecause 

mnai  operaoorrai  capaoiiity  win  not  occur  unDi  satenites  *rom  a  ruture  increment  are  tteioed. 

Gray  Eagle  UAV 


Program  Essentials 

Program  Performance  (fiscal  year  2012 

dollars  in  millions) 

Prime  contractor  General  Atomics 

As  of 

Latest 

Percent 

Aeronautical  Systems,  Inc. 

04/2005 

08/2011 

change 

Program  office:  Redstone  Arsenal,  AL 

Research  arto  development  cost 

$344.9 

$946.2 

174.4 

Funding  needed  to  complete: 

Procurement  cost 

$670.4 

$3,400.2 

4072 

R&D;  $226.1  ntillion 

Total  program  cost 

$1,015.2 

$5,158.9 

408.2 

Procurement;  $2,089.0  million 

Program  unit  cost 

$203,046 

$166-416 

-18.0 

Total  funding;  $3,006.9  million 

Total  quantities 

5 

31 

520.0 

Procurement  quantity:  16 

Acquisition  cycle  time  (months) 

50 

TBD 

NA 

TDtal  quartnes  induae  31  platoon  s«ls  wim  a  arcralt  eacn  TIte  program  will  also  Puy  21  alrcra*t  to 

replace  mose  lost  Dvougn  acnoon  and  7  training  aircran.  Hot  a  total  or  1 S2 

HC-130/MC  -130  Recapitalization  Program 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Prime  contractor  Lockheed  Martin 

As  of 

Latest 

Percent 

Program  office:  Wright-Patterson  AFB, 

03/2010 

12/2010 

change 

OH 

Research  and  development  cost 

$153.2 

$152.8 

-0.3 

Funding  needed  to  complete: 

Procurement  cost 

$7,699.3 

$12,621.9 

63.9 

R&D:  $82.6  million 

Total  program  cost 

$8,364.2 

$13,090,8 

56.5 

Procurement:  $9,532.4  million 

Program  unit  cost 

$113,029 

$107,302 

-5.1 

Total  funding;  $9,812.7  million 

Total  quantities 

74 

122 

64.9 

Procurement  quantity;  91 

Acquisition  cycle  time  (months) 

NA 

NA 

NA 

Contract  Number:  H98230-08-D-0171 


Report: No.  SERC-2013-1R-040-3  Volume 
November  12,  2013 

43 


TO  0030,  RT040 


UNCLASSIFIED 


IDECOM  Block  4 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  ITT  Electronic 

As  of 

Latest 

Percent 

Systems 

06f2008 

10/2011 

change 

Program  office:  Patuxent  River.  MD 

Research  and  development  cost 

$220.2 

$252.0 

14.5 

Funding  needed  to  complete: 

Procurement  cost 

$4742 

$569.5 

20.1 

R&D:S121.4  million 

Total  program  cost 

$694.4 

$821.5 

18.3 

Procurement:  S569.5  million 

Program  unit  cost 

$4,340 

$4,324 

-0.4 

Total  funding:  $690.9  million 

Total  quantities 

160 

190 

18.8 

Procurement  quantity;  190 

Acquisition  cycle  time  (months) 

59 

54 

-8.5 

Joint  High-Speed  Vessel 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Prime  contractor  Austal  USA 

As  of 

Latest 

Percent 

Program  office:  Washington,  DC 

02/2009 

12/2010 

change 

Funding  needed  to  complete: 

Research  and  development  cost 

$128.4 

$138.0 

7.4 

R&D;  $23.0  million 

Procurement  cost 

$3,507.9 

$3,536.1 

0.8 

Procurement;  $2,202.8  million 

Total  program  cost 

$3,636.4 

$3,674.1 

1.0 

Total  fundirtg;  $2,225.9  million 

Program  unit  cost 

$202,020 

$204,116 

1.0 

Procurement  quantity:  11 

Total  quantities 

18 

18 

0.0 

Acquisition  cycle  time  (months) 

48 

50 

4.2 

Joint  Land  Attack  Cruise  Missile  Defense 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  Raytheon 

As  of 

Latest 

Percent 

Program  office;  Redstone  Arsenal,  AL 

08/2005 

12/2010 

change 

Funding  needed  to  complete; 

Research  and  development  cost 

$2,005.5 

$2,523.2 

25.8 

R&D:  $634.1  million 

Procurement  cost 

$4,588.7 

$5,199.4 

13.3 

Procurement;  $5,199.4  million 

Total  program  cost 

$6,665.9 

$7,857.8 

17.9 

Total  funding;  $5,948.7  million 

program  unit  cost 

$416,619 

$491,112 

17.9 

Procurement  quantity:  14 

Total  quantities 

16 

16 

0.0 

Acquisition  cycle  time  (months) 

97 

103 

6.2 

Contract  Number:  H98230-08-D-0171 
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Joint  Precision  Approach  and  Landing  System 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  m 

illions) 

Prime  contractor  Raytheon 

As  of 

Latest 

Percent 

Program  office:  Lexington  Park,  MD 

07/2008 

08/2011 

change 

Funding  needed  to  complete: 

Research  and  development  cost 

$792.1 

$753.5 

.4.9 

R&D:  S183.9  rrallion 

Procurement  cost 

$213.2 

$222.7 

4.4 

Procurement:  $222.7  million 

Total  program  cost 

$1,012.3 

$983.3 

-2.9 

Total  furtding:  $406.6  million 

Program  unit  cost 

$27,359 

$26,575 

-2.9 

Procurement  quantity:  26 

Total  quantities 

37 

37 

0.0 

Acquisition  cycle  time  (months) 

75 

77 

2.7 

Airborne  and  Maritime  Joint  Tactical  Radio  System 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Prime  contractor  Lockheed  Martin 

As  of 

Latest 

Percent 

Program  office:  San  Diego.  CA 

10/2008 

08/2011 

change 

Funding  needed  to  complete: 

Research  and  development  cost 

$1,945.0 

$1,957.0 

0.6 

R&D;  $593.7  million 

Procurement  cost 

$6,209.0 

$6,203.8 

-0.1 

Procurement:  $6,203.8  million 

Total  program  cost 

$8,154.1 

$8,160.8 

0.1 

Total  funding:  $6,797.5  million 

F^rogram  unit  cost 

$.301 

$.301 

0.1 

Procurement  quantity:  26,878 

Total  quantities 

27.102 

27,102 

0.0 

Acquisition  cycle  lime  (months) 

80 

91 

13.8 

Trie  program  omce  reported  quanttiee  In  terms  or  cfianneis  ratner  than  radios 

Joint  Tactical  Radio  System  Handheld 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  General  Dynamics  C4 

As  of 

Latest 

Percent 

Systems,  Inc. 

05/2004 

11/2011 

change 

Program  office:  San  Diego.  CA 

Research  and  development  cost 

$544.7 

$1,272.3 

133.6 

Funding  needed  to  complete; 

Procurement  cost 

$9,492.8 

$7,085.7 

-25.4 

R&D;  $352.5  million 

Total  program  cost 

$10,037.5 

$8,357.9 

-16.7 

Procurement;  $7,022.1  million 

Program  unit  cost 

$.031 

$.031 

1.0 

Total  funding:  $7,374.6  million 

Total  quantities 

328,674 

270,951 

-17.6 

Procurement  quantity:  264,019 

Acquisition  cycle  lime  (months) 

85 

104 

22.4 

Contract  Number:  H98230-08-D-0171 
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LHA  Replacement  Amphibious  Assault  Ship 


Program  Essentials 

Program  Performance  (fiscal  year  2012 

dollars  in  millions) 

Prime  contractor  Huntington  Ingalls 

As  of 

Latest 

Percent 

Industries 

01/2006 

12/2010 

change 

Program  office:  Washington.  DC 

Research  and  development  cost 

$220.9 

$350.9 

58.8 

Funding  needed  to  complete: 

Procurement  cost 

$2,959.2 

$9,742.8 

229.2 

R&D:  $97,3  million 

Total  program  cost 

$3,180.2 

$10,095.2 

217.4 

Procurement:  $5,627.9  million 

Program  unit  cost 

$3,180,150 

$3,365,053 

5.8 

Total  funding:  $5,726.2  million 

Total  quantities 

1 

3 

200.0 

Procurement  quantity:  1 

Acquisition  cycle  time  (months) 

146 

165 

13.0 

Littoral  Combat  Ship 


Program  Essentials 

Program  Performance  (fiscal  year  2012 

dollars  in  millions) 

Prime  contractor  Austal  USA,  General 

As  of 

Latest 

Percent 

Dynamics,  Lockheed  Martin 

05/2004 

12/2010 

change 

Program  office:  Washington.  DC 

Research  and  development  cost 

$887.0 

$3,520.1 

296.9 

Funding  needed  to  complete: 

Procurement  cost 

$471.6 

$29,136.1 

6,078.2 

R&D:  $1,112.5  million 

Total  program  cost 

$1,358.6 

$32,867.8 

2,319.3 

Procurement:  $25,001.1  million 

Program  unit  cost 

$339.6 

$597,596 

76.0 

Total  funding;  $26,325.2  million 

Total  quantities 

4 

55 

1,275.0 

Procurement  quantity:  47 

Acquisition  cycle  time  (months) 

41 

116 

182.9 

Cost  daa  are  rorme  seaframe  only  me  2004  estinate  corresponos  vroi  program  initiation  it  was 

pre-milestone  B  ana  aia  not  reflect  3ie  tUll  55-sNp  program,  ‘tesearcn  ana  aeveiopmert  tunoing 

ncliioes  detai  design  and  construction  or  two  steps. 

Mobile  User  Objective  System 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  m 

illions) 

Prime  contractor  Lockheed  Martin 

As  of 

Latest 

Percent 

Space  Systems 

09/2004 

08/2011 

change 

Program  office:  San  Diego.  CA 

Research  and  development  cost 

$3,647.7 

$4,218.3 

15.6 

Funding  needed  to  complete: 

Procurement  cost 

$3,035.0 

$2,694.3 

-11.2 

R&D:  $470.1  million 

Total  program  cost 

$6,721.3 

$6,978.2 

3.8 

Procurement:  $1,125.1  million 

Program  unit  cost 

$1,120,222 

$1,163,035 

3.8 

Total  funding;  $1,595.2  million 

Total  quantities 

6 

6 

0.0 

Procurement  quantity:  1 

Acquisition  cycle  time  (months) 

90 

116 

28.9 

me  ales:  cost  data  do  not  reflect  die  current  cost  or  me  program.  A  new  acquisition  program  Paseilne 

nd6  rxK  yet  Deen  approved. 
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MQ-4C  BAMS  UAV 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Prime  contractor  Northrop  Grumman 

As  of 

Latest 

Percent 

Systems  Corporation 

02/2009 

12/2010 

change 

Program  office:  Patuxent  River,  MD 

Research  and  development  cost 

$3,141.7 

$3,245.6 

3.3 

Fundirtg  needed  to  complete: 

Procurement  cost 

$9,323.4 

$9,413.9 

1.0 

R&D:  $1,657.1  million 

Total  program  cost 

$12,847.6 

$13,052.4 

1.6 

Procurerrrent:  $9,413.9  million 

Program  unit  cost 

$183,537 

$186,463 

1.6 

Total  funding:  $11,422.2  million 

Total  quantities 

70 

70 

0.0 

Procurement  quantity:  65 

Acquisition  cycle  time  (months) 

92 

92 

0.0 

Navy  Multi-band  Terminal 

Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Prime  contractor  Raytheon 

As  of 

Latest 

Percent 

Program  office:  San  Diego.  CA 

12/2006 

08/2011 

change 

Fundirrg  needed  to  complete: 

Research  and  development  cost 

$697.2 

$666.2 

-4.4 

R&D:  $41.1  million 

Procurement  cost 

$1,623.7 

$1,214.4 

-25.2 

Procurement:  $992.6  million 

Total  program  cost 

$2,321.0 

$1,880.7 

-19.0 

Total  funding:  $1,033.8  milion 

Program  unit  cost 

$6,970 

$6,186 

-11.2 

Procurement  quantity:  189 

Total  quantities 

333 

304 

-8.7 

Acquisition  cycle  time  (months) 

107 

107 

0.0 

P-8A  Poseidon 

Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  millions) 

Prime  contractor  Boeing 

As  of 

Latest 

Percent 

Program  office:  Patuxent  River.  MD 

05/2004 

08/2011 

change 

Funding  needed  to  complete: 

Research  and  development  cost 

$7,531.5 

$8,215.3 

9.1 

R&D;  $1,232.4  million 

Procurement  cost 

$23,365.2 

$24,157.2 

3,4 

Procurement:  $20,087.9  million 

Total  program  cost 

$31,034.3 

$32,969.3 

6.2 

Total  furtding;  $21,839.0  million 

Program  unit  cost 

$269,864 

$270,240 

0.1 

Procurement  quantity:  104 

Total  quantities 

115 

122 

6.1 

Acquisition  cycle  time  (months) 

160 

160 

0.0 

Contract  Number:  H98230-08-D-0171 
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Reaper  UAV 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in  m 

illions) 

Prime  contractor  General  Atomics 

As  of 

Latest 

Percent 

Aeronautical  Systerrrs.  Inc. 

02/2008 

08/2011 

change 

Program  office:  Wright-Patterson  AFB, 

Research  and  development  cost 

$420.1 

$920.3 

119.1 

OH 

Procurement  cost 

$2,111.5 

$10,848.3 

413.8 

Funding  needed  to  complete: 

Total  program  cost 

$2,637.1 

$11,918.7 

352.0 

R&D:  $420.5  million 

Program  unit  cost 

$25,115 

$29,871 

18.9 

Procurement:  $7,962.6  million 

Total  quantities 

105 

399 

280.0 

Total  fundirrg:  $8,473.5  million 

Acquisition  cycle  time  (months) 

79 

94 

19.0 

Procurement  quantity:  240 

Space-based  Infrared  System  Program 


Program  Essentials 

Program  Performance  (fiscal  year  2012  dollars  in 

millions) 

Prime  contractor  Lockheed  Martin 

As  of 

Latest 

Percent 

Program  office:  B  Segundo,  CA 

10/1996 

07/2011 

change 

Funding  needed  to  complete: 

Research  and  development  cost 

$4,376.3 

$11,586.3 

164.7 

R&D:  $2,131.3  nvIMon 

Procurement  cost 

$0.0 

$6,429.3 

NA 

Procurement:  $3,599.4  million 

Total  program  cost 

$4,596.5 

$18,266.7 

297.4 

Total  furxling:  $5,743.9  million 

Program  unit  cost 

$919,301 

$3,044,443 

231.2 

Procurement  quantity:  2 

Total  quantities 

5 

6 

20.0 

Acquisition  cycle  time  (months) 

86 

TBD 

NA 

Tne  iMe  3313  snow  no  procurement  cost  3S  sie  AT  Force  pi3nne<l  lo  use  reseorcn  sno  aeve<opment 

runos  to  Ouy  3ll  five  S3teliites.  Tne  cost  or  me  two  meo  repienisivTient  sensors  is  not  induoed  in  eimer 

column. 

Standard  Missile  6  ERAM 


Program  Performance  (fiscal  year  2012  dollars  in  millions) 


Program  Essentials 

Prime  contractor  Raytheon  Missile 

Systems 

Program  office;  Arlington,  VA 
Funding  needed  to  complete: 

R&D;  S7.6  million 
Procurement:  S4, 808.9  million 
Total  funding:  $4,816.6  million 
Procurement  quantity:  1,111 


Research  arxj  development  cost 
Procurement  cost 
Total  program  cost 
Program  unit  cost 
Total  quantities 

Acquisition  cycle  time  (months) 


As  of 

Latest 

Percent 

07/2004 

12/2010 

change 

$1,073.8 

$973.5 

-9.3 

$4,626.4 

$5,323.2 

15.1 

$5,700.2 

$6,296.7 

10.5 

$4,750 

$5,247 

10.5 

1,200 

1.200 

0.0 

75 

94 

25.3 
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IV.  Next  STEPS 


The  most  pressing  need  in  this  research  is  access  to  information  on  completed  programs  that  will  help 
characterize  the  connection  between  some  definitions  of  complexity  and  the  post  hoc 
prediction/realization  of  technical  risk. 


In  addition,  we  shall  pursue  more  case  studies,  more  literature,  and  more  methods  of  characterizing 
complexity  of  products  and  organizations,  including  interviewing  acknowledged  experts. 
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Appendix  A:  Literature  Review  on  System  Complexity  and  Risk 


This  section  provides  a  summary  and  synthesis  of  findings  from  a  survey  of  the  iiterature  on  system 
compiexity  and  cost,  scheduie  and  performance  risk  in  engineering  deveiopment  programs.  Severai 
reievant  and  exempiar  papers  discussed  in  detaii. 

The  survey  was  restricted  to  pubiicaiiy-accessibie,  freeiy-avaiiabie  documents  via  the  internet  found  by 
searching  on  combinations  of  the  keywords  "compiexity",  "compiex",  "compiicated",  "risk", 
"uncertainty",  "overrun",  "siip",  "shortfaii",  "cost",  "scheduie",  "performance",  "acquisition", 
"deveiopment",  "engineering",  "engineered",  "technicai",  "quantitative",  "indicators",  "factors", 
"moduiar",  "adaptive",  "adaptabie",  "system",  "modei",  "architecture",  "anaiysis",  and  "assessment". 
From  over  2,000  web  sites  visited,  approximateiy  600  articies  were  reviewed.  Roughiy  three-quarters 
focused  on  risk  in  system  acquisition,  with  some  reference  to  system  compiexity.  The  remaining  quarter 
focused  on  compiexity  of  engineered  systems,  with  some  reference  to  deveiopment. 


A.l  Summary  OF  Findings 

This  section  summarizes  both  what  was  found  in  the  iiterature,  and  what  was  notabie  by  its  absence. 

Many  papers  asserted  that  increased  compiexity  was  correiated  with  increased  deveiopment  time  and 
cost.  Data  and  evidence  supporting  this  intuitive  ciaim  was  sparse.  Most  of  the  papers  were  theoreticai, 
often  using  a  "toy"  system  modei  to  iiiustrate  the  methods,  but  did  not  provide  evidence  that  their 
compiexity  metrics 

(1)  couid  be  evaiuated  from  data  on  DoD  systems  avaiiabie  during  their  deveiopment, 

(2)  were  good  predictors  of  deveiopment  time  and  cost,  or 

(3)  were  predictors  of  cost  increase,  scheduie  siip  or  performance  shortfaii. 

Of  the  haif-dozen  articies  that  appiied  compiexity  metrics  to  acquisition  program  data  and  anaiyzed  the 
reiationship  to  cost  and  scheduie,  there  were  oniy  three  distinct  anaiyses.  A  series  of  papers  by  the 
same  authors  anaiyzed  from  different  perspectives  one  set  of  Major  Defense  Acquisition  Program 
(MDAP)  data  provided  by  OSD.  The  finai  report  by  the  Boeing  team  on  the  DARPA  META  ii  program 
anaiyzed  data  from  two  different  divisions  of  the  Boeing  Corporation.  A  PhD  thesis  out  of  the  Air  Force 
institute  of  Technoiogy  anaiyzed  an  aircraft  avionics  data  set.  These  papers  are  reviewed  in  detaii. 

The  papers  on  compiexity  addressed  the  compiexity  of  the  system  design,  but  did  not  specify  the 
appropriate  ievei  of  architecture  and  design  data  for  anaiysis.  This  begs  the  question  of  whether  the 
design  information  is  avaiiabie  eariy  enough  in  the  acquisition  process  to  guide  architecture  and  design 
decision.  The  papers  using  the  MDAP  data  provided  by  OSD,  used  Systems  Engineering  artifacts 
produced  at  Miiestone  B. 

Many  of  the  articies  on  acquisition  risk  identified  the  turbuience  in  the  system  requirements  and 
interdependencies  among  the  requirements  (either  antagonistic  or  synergistic)  as  sources  of  deiay,  cost 
increase,  time  and  cost  uncertainty,  it  is  possibie  that  compiexity  anaiysis  couid  be  appiied  to  the 
network  of  requirements  (or,  more  generaiiy  the  system  baseiine,  which  consists  of  the  capabiiity  needs, 
the  system  requirements,  system  functionai  decomposition  and  requirements),  and  to  the  change  in  the 
baseiine  over  time.  This  might  provide  usefui  and  timeiy  insights  into  acquisition  risk  and  compiexity  of 
the  system  baseiine.  These  data  are  avaiiabie  to  the  acquisition  Program  Manager's  Office,  are 
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developed  over  time,  and  could  potentially  benefit  from  feedback.  No  analyses  of  requirements  or 
system  baseline  complexity  and/or  change  in  complexity  were  found  in  the  literature. 

The  term  "complexity"  was  not  used  consistently  throughout  the  literature.  Many  of  the  papers, 
especially  those  focused  primarily  on  development  risk,  used  the  terms  "complex"  and  "complicated" 
interchangeably,  generally  meaning  "systems  with  lots  distinct  parts  and  lots  of  connections  among  the 
parts."  The  papers  focusing  primarily  on  the  complexity  of  engineered  systems  used  a  variety  of 
descriptions  and  definitions  of  complexity.  A  number  of  the  papers  distinguished  between  "structural 
complexity"  and  "dynamic  complexity". 

"Structural  complexity"  was  used  to  refer  to  complexity  in  the  architecture  of  the  system.  Structural 
complexity  was  commonly  based  on  a  graph  of  nodes  (processors)  and  links  (interfaces)  representing 
the  system  architecture.  Metrics  ranged  from  simple  counts  of  the  number  of  links  and  nodes  (similar  to 
the  measures  of  "complicated"  systems),  to  refinements  using  algebraic  network  analysis  of 
interconnectedness.  Some  of  the  approaches  distinguished  between  the  number  of  instances  of  a  type 
of  node  and  the  number  of  distinct  types  of  nodes.  Some  distinguished  between  uni-directional  and  bi¬ 
directional  interfaces. 

"Dynamic  complexity"  refers  to  the  behavior  or  response  of  the  system  (e.g.,  states  and  transitions). 

The  term  was  used  variously  to  refer  to  systems  exhibiting  adaptive  response  to  external  states,  non¬ 
linear  change  in  response  depending  on  internal  state,  adaptive  response  to  internal  states,  self¬ 
organization,  cascading  effects,  unexpected  responses,  or  "emergent  behavior".  The  dynamic 
complexity  metrics  require  a  model  of  the  system  behavior  and  response.  Many  of  the  papers 
discussing  dynamic  complexity  did  not  present  computational  metrics.  Some  papers  used  a  system  state 
transition  diagram  model  of  the  system  dynamics,  then  applied  analysis  methods  similar  to  those  used 
for  architecture  structure  graph  complexity. 

Some  of  the  papers  distinguished  between  observable  complexity  in  the  system  model,  and  hidden 
complexity  within  nodes  and  links.  Some  of  the  papers  distinguished  between  complexity  inherent  in 
the  system  design,  and  apparent  complexity  due  to  incomplete  models,  incomplete  analysis,  and 
incomplete  characterization  of  the  boundary  conditions.  These  distinctions  are  related  to  the 
unresolved  issue  of  selecting  the  granularity  or  level  of  resolution  of  the  system  description,  i.e.,  the 
level  or  scope  of  components  or  subsystem  to  represent  as  distinct  nodes.  There  is  a  tradeoff  between 
the  size  of  the  model,  and  risk  of  errors  in  the  model,  versus  errors  due  to  ignorance  of  the  behavior, 
response,  and  internal  states  of  nodes  and  links.  None  provided  guidelines  for  determining  the 
appropriate  level  of  granularity  for  analysis. 

State  transition  diagrams  to  analyze  dynamic  complexity  suffer  from  combinatorial  explosion.  A 
network  with  5  nodes  and  5  links,  where  each  node  and  link  has  two  possible  states  (e.g.,  busy  and  not 
busy,  or  operative  and  not  operative,  at  capacity  or  below  capacity),  has  1024  possible  states  (2  to  the 
10*^  power),  and  over  1,000,000  possible  state  transitions  (1024  squared  minus  1024).  Complexity 
analysis  requires  determining  which  states  the  system  can  actually  occupy,  and  which  transitions  it  can 
experience.  Reducing  the  level  of  granularity  to  limit  the  size  of  the  model  increases  the  "hidden" 
complexity. 

Most  of  the  complexity  metrics  relied  on  a  node-and-link  graph  representations  of  the  system  in  which 
nodes  perform  processes,  and  interfaces  exchange  data,  energy,  material,  physical  position,  etc. 
between  nodes.  Processors  and  interfaces  have  performance  properties  beyond  being  logical  nodes  and 
links  in  a  graph:  capacity,  latency,  noise,  losses,  etc.  When  there  is  sufficient  design  margin  (reserve 
capacity)  so  that  there  is  negligible  risk  that  the  component  or  interface  is  overloaded  or  non-operative. 
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then  it  may  be  possible  to  omit  the  node  or  link  and  its  internal  states  from  the  analysis.  When  there  is 
non-negligible  risk,  then  the  node  or  link  and  its  internal  states  should  be  included  in  the  analysis. 

An  important  class  of  interfaces  not  addressed  in  the  papers  is  insulators  and  isolators  whose  purpose  is 
to  prevent  exchange  offeree,  energy,  signals,  etc.  between  nodes  and  between  links  (e.g.,  prevent  cross¬ 
talk,  short-circuits,  vibration  transfer,  thermal  degradation,  etc.).  Failure  of  insulators  and  isolators,  or 
temporary  failure  when  they  reach  their  excursion  limit,  create  short  circuits  that  can  radically  change 
the  response  of  the  system. 

A  widely  used  approach  to  compute  graph  complexity  involved  the  "graph  energy",  computed  from  the 
eigenvalues  of  the  adjacency  matrix  representing  the  graph  of  either  the  architecture  structure  or  the 
state  transition  diagram.  Closed-form  calculation  of  the  graph  energy  is  only  possible  for  simple  graphs, 
e.g.,  trees  structures  without  loops  or  lattice  structures.  Research  on  robust  methods  to  estimate  the 
approximate  graph  energy  for  arbitrary  graphs  is  an  area  of  on-going  research. 

A  less  widely  adopted  approach  to  complexity  was  to  use  a  measure  of  the  information  content  in  a 
minimal,  irreducible,  specification  of  the  system  model. 

Axiomatic  Design  provides  an  alternative  view  of  complexity  in  terms  of  the  probability  that  the  system 
can  perform  the  functions  required  of  it  at  any  given  time.  Axiomatic  Design  focusses  on  the 
relationship  between  system  structure  and  functions.  In  principle,  it  addresses  the  frequency  and 
duration  of  functions,  and  multiple  simultaneous  functions. 

Axiomatic  Design  suggests  approaches  to  understand  the  modularity  inherent  in  a  design,  either  by 
modules  associated  with  overlapping  functions,  or  block-diagonal  modularization  to  minimize  interfaces 
between  blocks.  Several  papers  contained  analytic  approaches  or  metrics  incorporating  modularity  into 
the  complexity  metrics.  There  is  a  small  body  of  literature  on  multi-scale  complexity,  but  it  is  oriented 
towards  biological  organisms,  not  engineered  systems. 


A.2  Reviews  of  Selected  Papers  and  Presentations 

This  section  contains  reviews  of  all  of  the  papers  and  presentations  analyzing  the  relationship  of 
complexity  to  cost  and  schedule  performance  (beginning  with  the  three  analyzing  the  OSD  MDAP  data 
set),  followed  by  reviews  of  selected  papers  exemplify  major  issues  and  approaches  in  complexity 
metrics  for  engineered  systems.  The  reviews  are  organized  under  the  following  topics: 

•  Systems  complexity  and  development  risk 

•  Structural  and  dynamic  complexity  metrics  from  graph  complexity 

•  Modularity  considerations  and  metrics  in  system  complexity 

•  Axiomatic  Design  approach  to  complexity 

•  Functional  and  contextual  complexity 

•  Apparent  complexity  in  flight  software 

•  Adaptability  metrics  to  measure  complexity 

•  Aspects  of  complexity  in  design 
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A.2.1  System  C  OMPLExnv  AND  DEVEiDPMEm' Risk 

Programmatic  and  Constructive  Interdependence:  Emerpinp  Insights  and  Predictive  Indicators  of 

Development  Resource  Demand,  M.  Kasunic,  M.M.  Brown,  PI.  Hardin,  J.M.  McCurley,  2010 

http://repositorv.cmu.edu/cgi/viewcontent.cgi?article=1008&context=sei 

Kasunic  et  al  [1]  describe  a  series  of  research  efforts  investigating  the  role  of  interdependence  in  the 
acquisition  of  Major  Defense  Acquisition  Programs  (MDAPs).  The  research  initiative  was  sponsored  by 
the  Office  of  the  Secretary  of  Defense  (OSD).  The  overall  goal  of  the  research  was  to  identify,  quantify, 
and  assess  the  degree  of  programmatic  and  constructive  interdependence  and  to  assess  the  effects  of 
interdependence  on  program  risk.  The  report  summarizes  the  results  of  five  research  studies  that  were 
conducted  from  2004  to  2009. 

Study  1  explored  the  qualitative  factors  that  confound  program  cost  and  schedule  estimation.  The  study 
identified  specific  risk  indicators  related  to  requirements,  institutional  factors,  sustainment,  and  team 
performance. 

Study  2  employed  data-mining  and  statistical  analyses  to  determine  whether  Defense  Acquisition 
Executive  Summary  (DAES)  reports  and  Select  Acquisition  Reports  (SARs)  can  be  used  to  forecast 
program  performance.  An  interesting  result  from  this  study  is  that  there  was  no  evidence  that  such 
indicators  are  effective  in  predicting  program  breaches. 

Studies  3-5  employed  network  analysis  techniques  to  quantitatively  characterize  programmatic  and 
constructive  interdependencies  in  the  acquisition  enterprise.  These  last  three  studies  culminated  in 
graphical  models  that  relate  interdependence  and  program  cost.  The  research  study  found  no  evidence 
that  indicators  reported  within  DAES  reports  or  SARs  predict  program  breach  events. 

Simple  Parametric  Model  For  Estimating  Development  (RDT&E)  Costs  for  Large-Scale  Systems,  R.R.  Jones, 
P.  Hardin,  A.  Irvine,  2005 

http://www.technomics.net/files/downloads/papers/ISPASCEA0609-Parametric.pdf 

Jones,  Hardin  and  Irvine  [2]  analyzed  data  provided  by  OSD(AT&L).  The  sponsor  provided  data  on  21 
Major  Defense  Acquisition  Programs  (MDAPs).  The  data  included  the  initial  RDT&E  cost  estimates  from 
Selected  Acquisition  Reports  (SARs),  and  architecture  structure  metrics  calculated  from  DoDAF  SV-6 
architecture  views:  numbers  of  send-only  nodes,  receive-only  nodes,  send-and-receive  nodes,  all  nodes, 
one-way  links,  two-way  links,  and  all  links.  The  analysis  of  the  relationship  between  the  total  number  of 
nodes  and  the  total  number  of  links  showed  a  strong  linear  correlation,  with  one  noticeable  outlier.  The 
analysis  initial  RDT&E  cost  showed  a  strong  correlation  with  the  square  of  the  number  of  links.  The  data 
clustered  into  three  groups  (a  large  number  of  data  points  with  low  cost  and  low  number  of  links,  three 
points  with  mid-range  cost  and  number  of  links,  and  one  point  with  high  cost  and  number  of  links),  so 
from  a  statistical  analysis  view,  there  were  only  three  distinct  data  points.  The  authors  found  a 
complicated  non-linear  formula  relating  cost  to  the  architecture  structure  metrics  with  almost  perfect 
correlation.  However  the  number  of  implicit  and  explicit  parameter  exceeded  the  statistically  significant 
degrees  of  freedom  in  the  data  set  as  analyzed.  Regardless  of  the  statistical  details,  the  report  presents 
system  architecture  metrics  derived  from  required  artifacts  (DoDAF  SV-6  architecture  is  required  prior  to 
Milestone  B),  with  strong  correlation  to  RDT&E  cost. 

The  sponsoring  agency  was  kind  enough  to  provide  the  SERC  with  the  original  data,  updated  to  include 
the  RDT&E  cost  estimates  as  of  2008.  We  re-analyzed  the  data,  with  care  not  to  "over-fit."  We 
conducted  a  bootstrap  analysis  (replicated  random  partitions  of  the  data  into  "training/calibration"  and 
"test/evaluation"  data  sets).  Analysis  in  log-log  space  showed  a  strong  linear  correlation  between  the 
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initial  RDT&E  cost  estimates  and  the  number  of  links  in  the  DoDAF  SV-6  diagrams.  In  log-log  space,  the 
RDT&E  cost  estimates  and  the  number  of  links  were  nicely  distributed  over  their  range.  The  dispersion 
about  the  linear  fit  provided  an  estimate  in  the  uncertainty  (error)  between  initial  RDT&E  estimates  and 
predictions  from  the  number  of  links.  These  results  showed  that  70-percent  of  the  variance  in  the 
logarithm  of  the  initial  estimates  of  RDT&E  cost  in  one  data  set  was  predicted  by  (a)  the  logarithm  of  the 
number  of  links,  and  (b)  the  linear  relationship  between  the  logarithm  of  initial  RDT&E  cost  estimates 
and  logarithm  of  the  number  of  links  derived  from  a  sequestered  data  set. 

No  relationship  in  the  change  in  RDT&E  costs  from  the  initial  estimates  and  the  2008  RDT&E  cost 
estimate  bore  any  relationship  to  the  architecture  parameters.  Since  the  programs  were  started  at 
different  times  and  on  different  schedules,  the  programs  developments  from  inception  to  2008  were 
not  samples  from  the  same  population.  The  DoDAF  SV-6  diagrams  and  architecture  data  were  not 
updated. 

Programmatic  Complexity  &  Interdependence:  Emergina  Insights  and  Predictive  Indicators  of 

Development  Resource  Demand,  R.  Flowe,  M.  Brown,  P.L  Hardin,  2000 

http://acquisitionresearch.net/  files/FY2009/NPS-AM-09-058.pdf 

Flowe,  Brown  and  Flardin  [3]  prepared  report  for  the  Defense  Science  Board  addressing  the  effects  of 
technical  interdependence  among  programmatically-independent  acquisitions,  whether  explicit  as  in 
Systems-of-Systems,  or  implicit.  The  report  finds  that  interdependencies  have  non-linear  scaling  effects 
that  are  not  captured  in  technical  development  and  integration  cost  estimates.  The  research  used  data 
extracted  from  formal  program  documentation  including  Defense  Acquisition  Executive  Summary 
(DAES)  Charts,  Selected  Acquisition  Reports  (SARs),  Budget  Item  Justification  Exhibits,  Information 
Support  Plans  (ISPs),  etc.  The  research  analyzed  dependencies  of  MDAP  programs  on  other  MDAP 
programs  and  on  non-MDAP  programs  that  were  not  required  to  report  program  status.  The  report 
presented  a  network  diagram  program  interdependence  and  cost  growth  of  the  MDAPs,  but  did  not 
quantitatively  analyze  the  data. 

The  network  consisted  of  21  MDAPs,  162  non-MDAP  programs,  10  direct  dependencies  between 
MDAPS,  and  257  dependencies  of  MDAPs  on  non-MDAP  programs.  On  average,  an  MDAP  program  had 
13  external  dependencies  (one  to  another  MDAP  program,  and  12  to  non-MDAP  programs).  78-percent 
of  the  non-MDAP  programs  had  exactly  one  dependent  MDAP;  the  remaining  22-percent  had,  on 
average,  2.6  dependent  MDAPs. 

The  fourteen  MDAPs  with  less  than  50%  cost  growth  had  an  average  of  11  external  dependencies 
(sample  standard  deviation  of  4),  while  the  seven  MDAPs  with  more  than  50%  cost  growth  had  an 
average  of  17  external  dependencies  (sample  standard  deviation  of  7).  The  difference  suggests  that 
more  external  dependencies  was  correlated  with  greater  cost  overrun,  but  the  statistical  confidence  is 
low  due  to  the  large  variance. 

Impact  of  weapon  system  complexity  on  systems  acquisition,  R.A.  Dietrick,  2006 

http://dtlweb.au.af.miI///exlibris/dtl/d3  1/apache  media/L2V4bGlicmlzL2R0bC9kM18xL2FwYWNoZV9t 

ZWRpYS81MDk3Nw==.pdf 

In  his  PhD  thesis,  Dietrick  [4]  used  the  number  interactions  among  components  as  the  theoretical 
complexity  metric.  Interactions  include  space,  energy,  information,  and  material  exchange.  Due  to  the 
difficulty  in  counting  the  actual  interactions  among  system  components,  as  a  practical  metric  the  paper 
uses  the  theoretical  upper  bound  on  the  number  of  interactions  among  components  as  the  practical 
measure.  The  upper  bound  is  N(N-l)/2  where  N  is  the  number  of  components.  The  paper 
acknowledges  that  the  level  of  resolution  to  identify  components  will  affect  the  results,  and  comparison 
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across  systems  required  analysis  at  the  same  level  of  resolution.  It  does  not  consider  modularity  and 
hierarchical  organization. 

The  thesis  presents  empirical  data  for  USAF  aircraft  showing:  (1)  increase  of  complexity  with  increase  of 
year  of  operating  capability,  (2)  increase  of  development  time  with  year  of  operating  capability,  and  (3) 
increase  of  development  cost  with  increase  of  development  time.  The  thesis  does  not  directly  analyze 
system  development  time  or  cost  as  function  of  complexity  on  an  individual  system  basis.  The  thesis 
provides  an  analysis  of  trends,  not  an  analysis  of  systems.  The  paper  does  not  address  cost  increase, 
schedule  slip,  or  performance  shortfall. 

META  II  Complexity  and  Adaptability  Final  Report,  D.  Stuart,  R.  Mattikalli,  D.  DeLaurentis,  J.  Shah,  2011 

http://www.darpa.mil/uploadedFiles/Content/Our  Work/TTO/Programs/AVM/Boeing%20META%20Fin 

al%20Report.pdf 

The  Boeing  team's  final  report  on  complexity  and  adaptability  metrics  for  the  DARPA  META  program 
(Stuart  et  al  [5])  took  an  investigative,  opportunistic  and  integrated  approach.  They  did  not  pursue  a 
sequence  of  first  developing  a  theory  or  computational  model  of  complexity,  then  seeking  data  to 
evaluate  metrics,  then  seeking  data  on  cost  and  schedule,  then  testing  the  ability  of  the  model/metric  to 
explain  the  variance  in  cost  and  schedule.  Instead,  the  team  identified  28  already-defined  factors  with 
potential  value  in  a  complexity  metric,  that  could  be  evaluated  from  available  system  architecture  data. 
Calculation  methods  for  each  of  the  inputs  are  included  in  the  report.  The  identified  two  sources  of 
aircraft  program  data  (Boeing  Commercial  Aircraft,  BCA,  and  Boeing  Defense  Systems,  BDS)  for  which 
system  architecture  data,  cost  data  and  schedule  data  were  available  (peak  labor  was  used  as  a  proxy 
for  cost).  Armed  with  this  data,  the  team  conducted  a  "combinatorial  search"  for  combinations  of  input 
terms  models  to  explain  the  variances  in  peak  labor  and  in  schedule,  using  regression  to  estimate  the 
coefficients,  including  linear  and  logarithm  operations  on  the  inputs,  additive  and  multiplicative 
relationships.  The  models  that  were  finally  selected  contained  up  to  six  terms. 

The  report  noted  that  significant  manual  effort  was  required  to  extract  the  architecture  data  for  the  BDS 
programs,  including  interviews  with  the  chief  engineers.  The  BCS  data  were  extracted  from  an 
automated  project  management  system.  The  sample  size  was  approximately  15  BCA  projects  and  15 
BCD  projects.  The  modeling  was  conducted  separately  for  the  BCA  and  BCD  projects,  presumably 
because  the  team  suspected  differences  due  to  the  type  of  project  and  data  sources.  Not  only  did  the 
coefficients  for  the  models  differ  between  the  two  data  sets,  but  different  inputs  and  functional  forms 
ended  up  being  selected. 

"Combinatorial  search"  modeling  is  at  risk  of  using  up  the  degrees  of  freedom  in  the  data,  unless  the 
sample  size  is  large.  There  were  28  inputs,  with  the  option  of  taking  logarithm  or  squaring  for  each,  and 
the  options  of  mixed  addition  and  multiplication,  there  were  many  more  possible  models  than  the 
sample  size  supported.  The  report  acknowledges  the  sample  size  issue. 

The  team  could  have  used  bootstrap  or  similar  techniques  to  examine  the  stability  and  validity  of  the 
models.  They  did  not  randomly  divide  the  data  into  a  pair  of  disjoint  groups,  apply  the  process  to  the 
different  groups  to  test  if  the  same  input  parameters  were  selected  for  both  groups.  Ideally,  in  this 
model  development  approach,  iterative  replication  of  the  following  steps  are  used  to  develop  and  justify 
the  model:  (1)  divide  the  population  into  three  groups,  (2)  use  the  first  group  to  select  the  input  terms 
and  non-linear  functions  of  the  model,  (3)  use  the  second  group  to  estimate  the  values  of  the 
coefficients,  and  (4)  use  the  third  group  to  evaluate  the  explanatory  power  of  the  model.  This  process  is 
repeated  with  different  random  partitions  to  analyze  the  stability  and  validity  of  the  modeling  results. 
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A.2.2  SiRUCiURALAND  Dynamic  CoMPLExnv  Metric s from  Graph  Compiexfiy 

Meta  II  Complex  Systems  Design  and  Analysis  (CODA)  Final  Report,  B.  T.  Murray,  A.  Pinto,  R.  Skelding,  O. 
de  Week,  H.  Zhu,  S.  Nair,  N.  Shougarian,  K.  Sinha,  S.  Bopardikar,  and  L.  Zeidner,  2011 

http://www.dtic.mil/dtic/tr/fulltext/u2/a552676.pdf 

Murray  et  al  [6]  reported  on  the  United  Technologies  team  results  on  the  DARPA  META  II  program.  The 
report  covers  all  aspects  of  the  United  Technologies  effort  on  the  program.  Section  3.16,  "Complexity 
and  Adaptability  Metrics  in  Design"  is  particularly  relevant.  The  project  did  not  apply  the  methods  to 
real  systems  or  attempt  to  investigate  their  ability  to  explain  cost,  schedule,  overruns  and  slippage.  This 
paper  was  selected  to  review  as  an  exemplar  of  the  architecture  graph  analysis  methods. 

The  technical  approach  used  a  "graph"  model  of  the  system,  i.e.,  a  model  consisting  of  nodes  and 
interfaces  (information,  energy,  force,  momentum,  data,  signals,  fluids,  positional  relationship,  etc. 
exchanged  between  nodes).  For  computational  purposes,  the  graph  is  represented  by  the  Design 
Structure  Matrix  (DSM):  one  row  and  column  for  each  node,  a  one  in  the  matrix  if  there  is  an  interface 
from  the  row  node  to  the  column  node,  zero  otherwise,  and  zero  on  the  diagonal.  A  simplified,  non- 
directional  versions  of  the  DSM  is  the  association  matrix:  one  row  and  column  for  each  node,  a  one  in 
the  matrix  if  there  is  an  interface  in  either  direction  between  the  row  node  to  the  column  node,  zero 
otherwise,  and  zero  on  the  diagonal. 

Structural  complexity  refers  to  the  complexity  of  connections  between  subsystems.  The  proposed 
metric  was  the  number  of  components  plus  the  product  of  the  number  of  interfaces  times  the  "Graph 
Energy".  Graph  Energy  is  computed  from  the  adjacency  matrix,  a  non-directional  simplification  of  the 
DSM,  has  a  one  in  each  cell  if  the  row  node  and  column  node  have  an  interface  in  either  direction,  and 
zero  otherwise.  For  simple  graphs,  i.e.,  tree  structures  without  loops,  the  Graph  Energy  is  computed  as 
the  sum  of  the  absolute  values  of  the  eigenvalues  of  the  adjacency  matrix.  In  very  simple  geometries, 
loops  can  be  isolated  are  treated  as  a  single  node.  In  complex  geometries,  e.g.,  multiple  input  and 
output  nodes  within  loops,  nested  loops,  intersecting  loops,  etc.  other  computational  means  are 
needed,  such  as  Gibbs  sampling,  the  elimination  algorithm,  and  belief  propagation.  These  methods  are 
approximate  and  not  guaranteed  to  converge. 

Dynamic  complexity  refers  to  the  complexity  of  connections  between  transient  states  of  the  system. 
Instead  of  analyzing  the  links  between  nodes  of  the  system  architecture,  dynamic  entropy  examines 
links  between  states  of  the  system.  The  state  of  the  system  is  a  vector  with  an  element  for  each  node 
and  link.  Two  states  are  connected  if  there  is  a  single  event  that  will  cause  the  system  to  transition  from 
one  state  to  the  other.  Dynamic  complexity  is  computed  in  the  same  manner  as  structural  complexity, 
except  applied  to  the  state  association  matrix:  the  number  of  states  plus  the  number  of  state  transitions 
times  the  graph  energy  of  the  state  association  matrix. 

The  SVD  calculation  of  graph  energy  works  only  for  simple  graphs,  i.e.,  systems  without  feedback  loops. 
Other  methods  computational  methods  are  needed  to  estimate  the  graph  energy  for  arbitrary  graphs, 
and  specifically  for  systems  with  nested  and  intersecting  feedback  loops. 


A.2.3  MODULARTTY  CONSTDERAHONS  AND  METRICS  IN  SYSTEM  COMPIEXTIY 

Degree  of  Modularity  in  Engineering  Systems  and  Products  with  Technical  and  Business  Constraints,  K. 

Hoitta-Otto  and  O.  de  Week,  2007 

http://strategic.mit. edu/docs/2  19  CERA  15  2  113.pdf 
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Hoitta-Otto  and  de  Week  [7]  present  a  structural  modularity  metric  and  "packing  factor"  metric  to 
complement  structural  complexity  metrics  (such  as  the  number  of  nodes  plus  the  number  of  links  times 
the  graph  energy).  The  graph  energy  measures  the  total  system  connectivity.  The  goal  of  the 
modularity  index  is  to  measure  the  concentration  of  connectivity.  The  idea  behind  the  modularity  index 
is  that  important  information  describing  system  connectivity  is  concentrated  in  the  subset  of 
components  that  are  highly  connected  across  the  network.  The  modularity  index  is  an  attempt  to 
measure  connectivity  concentration. 

The  structural  modularity  metric  is  derived  from  the  SVD  of  the  adjacency  metric,  as  is  the  graph  energy. 
The  graph  energy  is  the  sum  of  the  absolution  values  in  the  SVD.  Since  the  SVD  is  a  diagonal  matrix,  it 
can  be  collapsed  to  a  vector,  and  sorted  in  decreasing  order  of  absolute  value.  The  authors  use 
exponential  decay  as  a  model  of  the  decrease  in  sorted  absolute  value  SVD  elements.  The  structural 
modularity  metric  is  derived  from  the  estimated  rate  of  decay.  Other  measures  of  concentration  that  do 
not  assume  an  exponential  decay  could  be  formulated  to  measure  the  concentration  of  magnitudes  in 
the  SVD.  The  authors  show  how  the  modularity  in  structural  modularity  metric  dex  discriminates  among 
several  architectures  with  equal  numbers  of  nodes,  links  and  graph  energy. 

Using  the  SVD  to  compute  the  structural  modularity  metric  has  the  same  drawback  as  using  the  SVD  to 
compute  graph  energy:  it  only  works  for  "simple"  graphs  -  tree  structures  without  loops.  The  authors 
do  not  propose  an  approach  to  compute  the  metric  for  arbitrary  graphs. 

The  authors  also  propose  "packing  density"  as  a  modularity  metric  to  complement  the  structural 
modularity  metric.  The  packing  density  metric  is  the  ratio  of  the  sum  of  the  volume  of  space  needed  for 
each  component  to  operate  (including  space  to  move,  as  appropriate)  divided  by  the  total  volume  of  the 
system.  A  similar  metric  could  be  generated  for  weight  and  other  cumulative  constraints.  The  packing 
density  metric  does  not  account  for  systems  that  share  space  (one  moves  out  as  the  other  moves  in; 
since  the  motion  is  coordinated,  presumably  these  are  considered  to  be  one  component). 


A.2.4  Axiomatic  Design  Approach  to  Compiexfiy 

Complexity  Theory  in  Axiomatic  Design,  T.  Lee,  2003 

http://citeseerx.ist. psu.edu/viewdoc/download?doi=10.1.1.135.4528&rep=repl&type=pdf 

Lee's  PhD  thesis  [8]  was  selected  for  review  because  it  presents  a  principled  approach  that  expands  the 
notion  of  complexity  as  a  function  of  the  structure  and  dynamics  of  a  system  to  include  the  functions  of 
the  system. 


The  complexity  concept  in  axiomatic  design  theory  is  defined  as  a  measure  of  the  likelihood  of  not 
achieving  a  desired  set  of  functional  requirements.  In  this  thesis,  four  different  types  of  complexity  are 
identified  in  axiomatic  design  complexity  theory:  time-independent  real  complexity,  time-independent 
imaginary  complexity,  time-dependent  combinatorial  complexity  and  time-dependent  periodic 
complexity.  Time-independent  real  complexity  is  equivalent  to  the  information  content,  which  is  a 
measure  of  a  probability  of  achieving  functional  requirements.  Time-independent  imaginary  complexity 
is  defined  as  the  uncertainty  due  to  ignorance  of  the  interactions  between  functional  requirements  and 
design  parameters.  Time-dependent  complexity  consists  of  combinatorial  complexity  and  periodic 
complexity,  depending  on  whether  the  uncertainty  increases  indefinitely  or  occasionally  stops  increasing 
at  certain  point  and  returns  to  the  initial  level  of  uncertainty.  In  this  thesis,  existing  definitions  for  each 
of  the  types  of  complexity  are  further  elaborated  with  a  focus  on  time-dependent  complexity.  In 
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particular,  time-dependent  complexity  is  clearly  defined  using  the  concepts  of  time-varying  system 
ranges  and  time-dependent  sets  of  functional  requirements. 


The  Axiomatic  Design  model  is  that  a  Design  Matrix  (DM)  specifies  how  the  Design  Parameters  (DP)  are 
related  to  the  Functional  Requirements  (FR).  The  FR  are  input  to  the  design  process,  the  DP  are  output. 
The  axioms  (or  principles)  of  Axiomatic  Design  are: 

•  Independence  Axiom:  Maintain  the  independence  of  functional  requirements 

•  Information  Axiom:  Minimize  the  information  content. 

Information  content  is  defined  as  the  negative  probability  of  achieving  the  functional  requirements  over 
the  range  of  conditions.  The  paper  defines  real  complexity  as  sensitivity  of  the  information  content  to 
changes  in  Functional  Requirements,  and  imaginary  complexity  as  uncertainty  in  DP  values,  due  to 
uncertainty  in  FR  and  DM. 


A.2.5  RjNCHONALANDCOISTTEXlUALCOMPlEXnY 

The  mathematics  of  IT  simplification,  R.  Sessions,  2011 

https://dl.dropboxusercontent.com/u/97323460/WebDocuments/WhitePapers/MathOflTSimplification- 

103.pdf 

Sessions  [9]  addresses  functional  complexity  (as  opposed  to  structural  or  dynamic  complexity).  The 
paper  suggests  that  functional  complexity  has  two  complementary  components:  internal  functional 
complexity  and  external  coordination  complexity.  The  goal  of  the  paper  is  to  develop  principles  to 
partition  a  system  to  simplify  development.  The  approach  is  inherently  hierarchical  and  can  be  applied 
to  hierarchical  partitioning  or  embedding  systems.  The  paper  does  not  explicitly  relate  the  metrics  to 
development  risk. 

In  principle  the  paper  takes  two  views  of  the  system:  as  a  stand-alone  system,  and  as  an  integral 
component  of  a  system-of-systems.  Internal  functional  complexity  is  a  measure  of  complexity  as  a 
stand-alone  system.  External  coordination  complexity  is  a  measure  of  complexity  of  the  role  in  a  system 
of  systems.  The  fundamental  difference  is  the  perspective,  not  the  computational  method.  Analogous 
methods  are  applied  in  both  perspectives. 

The  approach  computes  complexity  as  the  number  of  interactions  of  states  of  a  system  over  the  system 
functions.  The  number  of  states  of  a  system  computed  from  the  number  of  variables  (elements;  nodes), 
the  number  of  possible  states  for  each  node,  and  the  interdependencies  among  the  nodes  to  accomplish 
system  functions.  For  a  single  function,  it  disregards  all  nodes  whose  state  does  not  affect  the  function. 
It  determines  independent  groupings  according  to  the  rule  that  two  nodes  are  in  the  same  group  if  the 
performance  of  the  function  is  a  non-linear  function  of  the  two  nodes.  Within  a  grouping,  the  number 
of  states  is  the  product  of  the  number  of  states  pertaining  to  that  function  over  all  nodes  in  the  group. 
Different  functions  may  produce  some  of  the  same  groups.  The  measure  of  complexity  is  the  sum  over 
all  groups. 

Further  rationalization  in  defining  groupings  may  be  possible  (using  the  framework  of  normal  forms  in 
rational  database).  The  approach  does  not  analyze  overlap  between  functions.  Consider  two  groupings 
A  and  B  based  on  two  different  functions.  If  A  and  B  are  disjoint,  the  complexity  of  the  union  is  equal  to 
the  complexity  of  the  sum  (the  formulation  in  the  paper).  If  the  nodes  A  and  B  are  identical,  the 
combined  complexity  should  be  equal  distinct  states  over  both  function,  i.e.,  the  sum  of  the  individual 
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complexity  minus  the  number  of  overlapping  states.  If  A  and  B  partially  intersect,  the  complexity  is  the 
sum  of  the  individual  complexity  minus  the  number  of  overlapping  states. 


A.2.6  APFAREm'CoMPLExnYiN  RjghtSofiware 

NASA  study  on  Flipht  Software  Complexity,  D.L.  Dvorak,  2009 
http://www.nasa.gov/pdf/418878main  FSWC  Final  Report.pdf 

The  NASA  study  on  flight  software  complexity  [10]  was  driven  by  a  perception  that  flight  software  was  a 
major  source  of  cost  and  time  growth.  The  goal  of  the  study  was  to  identify  the  problems  and  find  ways 
to  mitigate  those  problems.  In  this  sense,  the  study  was  an  empirical,  investigative  study,  not  a 
theoretical  study.  It  did  not  develop  a  formal  complexity  metric,  but  instead  develop  a  list  of  potential 
causes  and  indicators  to  track.  The  study  identified  a  handful  of  software  complexity  metrics  in  the 
literature  (e.g.,  cyclomatic  complexity,  Halstead  complexity,  Henry  and  Kafura  metrics,  Bowles  metrics. 
Trot  and  Zweben  metrics,  Ligier  metrics),  but  did  not  devote  resources  to  the  study  of  complexity 
metrics  because  the  issues  they  identified  were  not  well  addressed  by  the  metrics. 


The  study  adopted  the  IEEE  Standard  Computer  Dictionary  definition  of  'complexity'  as  "the  degree  to 
which  a  system  or  component  has  a  design  or  implementation  that  is  difficult  to  understand  and  verify" 
The  phrase  "difficult  to  understand"  indicates  that  complexity  is  inherently  about  the  knowledge  and 
understanding  of  the  personnel  involved  in  the  project.  Complexity  in  this  sense  is  not  a  computable 
property  of  the  engineered  system.  But  this  sense  of  complexity  is  directly  related  to  the  likelihood  of 
inaccurate  estimates  and  mistaken  decisions. 


Flight  software  complexity  is  related  to: 

•  How  difficult  it  is  for  a  programmer  to  implement  the  requirements  the  code  must  satisfy? 

•  How  difficult  it  is  for  a  tester  to  verify  that  the  code  satisfies  the  requirements  and  operates  in  an 
error-free  fashion? 

•  How  difficult  it  is  for  a  lead  developer  to  manage  the  development  of  the  flight  software  within  cost 
and  schedule? 

•  How  difficult  it  is  for  a  flight  software  maintenance  programmer  to  understand  the  original 
programmer's  work  if  the  software  must  be  modified  after  launch? 

•  How  difficult  it  is  for  a  new  programmer  on  a  later  mission  to  adapt  the  original  flight  software  as 
heritage  for  the  new  mission? 

•  How  difficult  it  is  to  predict  the  flight  software's  behavior,  which  in  turn  can  drive  much  more 
extensive  testing  and  more  operational  "hand-holding"  along  with  their  associated  higher  labor  costs? 


Factors  used  to  measure  a  flight  software  system's  essential  complexity  include: 

•  How  many  functions  the  flight  software  must  execute  and  monitor? 

•  How  many  hardware  components  the  flight  software  must  monitor,  command,  control,  and  query  for 
information? 
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•  How  many  connections  (both  hardware  and  software)  between  components  the  flight  software  must 
monitor  and  manage? 

•  How  many  control  modes  must  be  managed  and  executed? 

•  How  many  software  modules  must  be  implemented  in  order  to  satisfy  the  flight  software's 
requirements? 

•  How  much  coupling  there  is  between  software  modules? 

•  How  intricate/convoluted  the  code  is  within  a  module  (assuming  best  programming  practices,  this  is  a 
measure  of  the  complexity  of  an  associated  requirement  or  algorithm  itself)? 

•  How  many  tests  must  be  created  and  executed  in  order  to  verify  that  the  flight  software  has  satisfied 
its  requirements  and,  in  fact,  whether  it  is  even  possible  given  limited  time  and  cost  to  verify  satisfaction 
of  those  requirements  under  all  likely  scenarios? 

•  How  "state-of-the-art"  the  requirement  is  (reflected  in  how  demanding  performance  and  accuracy 
requirements  are  relative  to  contemporary,  heritage  systems)? 


A.2.7  ADAPiABiuTir  Metric  SID  Measure  Complextiy 

Desipninp  Systems  for  Adaptability  by  Means  of  Architecture  Options,  A.  Engel  and  T.  R.  Browning,  2006 
http://www.incose.org/symp2008/dmdocuments/paper  example01.pdf 

In  [11],  Engel  and  Browning  assert  that  the  objectives  of  design  for  adaptability  are  to  make  complexity 
manageable,  to  enable  parallel  work,  to  enable  efficient  recovery  from  mistakes,  and  to  accommodate 
future  uncertainty.  Adaptability  mitigates  against  the  risk  of  changing  needs  and  technologies.  In  the 
sense  of  the  IEEE  Standard  Computer  Dictionary  definition  of  'complexity'  as  "the  degree  to  which  a 
system  or  component  has  a  design  or  implementation  that  is  difficult  to  understand  and  verify", 
adaptability  is  the  antithesis  of  complexity. 

The  paper  by  Engel  and  Browning  []  develops  static  and  dynamic  approaches  to  evaluate  adaptability.  It 
develops  metrics  for  six  dimensions  of  adaptability  (functionality,  reliability,  usability,  efficiency, 
maintainability,  and  portability).  Each  of  these  dimensions  is  further  subdivided.  The  paper  applies  real 
options  theory  to  assess  value  of  a  design  including  the  present  value  of  the  future  options  the  design 
provides. 


A.2.8  Aspects  OF  CoMPiExnY  IN  Design 

Complexity  models  in  design,  C.  Earl,  C.  Eckert,  and  J.  Johnson,  2004 
http://oro.open.ac.Uk/7420/l/Complexitv  Models  2004.pdf 

Earl,  Eckert  and  Johnson  [12]  distinguish  product  organization  complexity,  product  function  complexity, 
product  operational  complexity,  development  process  complexity,  and  development  organization 
complexity.  The  paper  describes  a  variety  of  aspects  of  complexity  including  structure,  uncertainty 
(knowledge  in  hand  versus  sufficient  knowledge  for  design,  evaluation  and  operation),  dynamic 
cascading  effects,  and  dynamic  adaptation.  The  goal  of  the  paper  is  to  advance  understanding  of  design 
processes  and  design  outcomes.  No  metrics  are  given.  No  direct  relationship  to  acquisition  risk  is 
presented. 
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Abstract 

In  this  research  we  compared  traditional  DoD  acquisition  risk  management  to  risk  management  in  other 
diverse  domains.  We  made  six  significant  observations  regarding  opportunities  to  improve  and  expand 
risk  management  in  DoD  system  acquisition.  We  formulated  a  framework  and  approach  for  methods, 
procedures  and  tools  (MPT)  to  enhance  the  traditional  DoD  characterization  of  risk,  to  provide  more 
rigorous  and  objective  methods  for  risk  assessment,  to  integrate  risk  analysis  with  program  decisions 
throughout  the  stages  of  acquisition  and  across  the  program  components,  to  identify  risk  areas,  and  to 
alert  program  management  for  the  needed  to  consider  implementing  risk  mitigations.  We  identified 
MPT  from  outside  traditional  DoD  acquisition  risk  management  that  could,  potentially,  be  adapted.  The 
next  phase  of  the  project  will  include  initial  development  of  preliminary  MPT,  pilot  testing  on  a  DoD 
program,  and  refinement  in  coordination  with  a  Service  agency  end-user  as  a  co-developer  and 
transition  partner  to  ensure  that  the  MPT  are  relevant  and  practical. 

Risks,  in  the  traditional  DoD  characterization  are  specific  future  events  that  have  some  probability  and 
consequence  that  can  be  estimated  in  advance.  An  expanded  characterization,  modeled  after  risk 
analysis  in  other  domains,  profiles  the  risk  propensity  and  risk  exposure  of  the  program  as  measured  by 
a  set  of  risk  indicators.  Decision  making  is  informed  by  assessing  the  impacts  of  alternative  decision 
choices  on  the  risk  profile.  Warning  thresholds  on  the  risk  indicators  alert  program  management  for  the 
need  to  consider  risk  mitigation.  Characteristic  program  risks,  indicators,  and  potential  mitigations  are 
linked  in  a  Risk  Breakdown  Structure  (RBS).  The  RBS  provides  a  framework  to  organize  rigorous  risk 
management. 
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Objectives  and  Approach 


The  objectives  of  this  phase  were  to  identify  opportunities  to  improve  risk  management  in  DoD 
acquisition  programs  by  comparing  the  traditional  DoD  approach  to  methods  and  approaches  in  other 
domains,  and  to  formulate  a  path  forward  for  the  next  phase  to  adapt  relevant  approaches,  to  develop 
and  implement  MPT,  to  pilot  test  and  refine  the  MPT  working  with  end-user  co-developer  transition 
partners. 

The  research  drew  on  documented  DoD  risk  management  guidance,  extensive  discussion  and  review  of 
risk  management  practices  with  our  end-user  co-developer  transition  partner,  US  Army  TARDEC,  and  on 
personal  experience  on  DoD  major  development  programs. 

The  research  drew  on  approaches  to  risk  assessment  and  mitigation  from  diverse  sources.  The  sources 
included  risk  management  in  commercial  engineering  development  projects,  commercial  engineering 
and  construction  project  financial  underwriting,  and  insurance  risk  adjustment.  The  sources  included 
non-DoD  Government  agencies  such  as  National  Aeronautics  and  Space  Administration  (NASA),  the 
Department  of  Energy  Nuclear  Regulatory  Commission  (DoE  NRC),  the  Environmental  Protection  Agency 
(EPA),  the  Federal  Aviation  Administration  (FAA),  and  the  Department  of  Flomeland  Security  (DFIS).  The 
diverse  domains  included  medical  care,  oil  and  gas  field  development,  nuclear  power  plant  construction, 
chemical  plant  construction,  satellite  launch  and  operation,  tunneling,  etc.  The  research  also  drew  on 
academic  and  applied  advances  in  areas  of  Systems  Engineering  outside  of  traditional  risk  management. 

We  compared  DoD  risk  management  practice  to  risk  management  in  other  domains,  and  made  six 
significant  observations  regarding  opportunities  to  develop  and  employ  MPT  for  improved  DoD  risk 
management.  We  also  identified  initial  sources  adapt  or  build  upon  to  develop  the  MPT.  Of  the  six 
observations,  the  first  four  lend  themselves  to  incremental  development  and  testing  with  the 
expectation  of  producing  preliminary  MPT  products  within  the  next  year,  acknowledging  the  possibility 
of  further  subsequent  refinement  and  extension.  The  last  two  of  the  six  observations  involve 
challenging  issues  in  cross-program  data  reporting,  collection,  analysis  and  dissemination.  Due  to  these 
challenges,  research  into  MPT  related  to  the  last  two  observations  will  be  deferred. 


Findings  and  Observations 


Observation  #1:  The  traditional  DoD  definition  and  characterization  of  risk  is  narrower  than  that  used  in 
other  domains.  The  DoD  definition  of  a  risk  is  "a  potential  future  event  with  adverse  consequences." 
This  definition  inhibits  recognition  of  various  types  and  sources  of  risk,  e.g.,  adverse  consequences  due 
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to  events  that  do  not  occur,  and  risks  with  multiple  compound  causes  but  no  single  event.  It  leads  to 
unreliable  assessments  of  the  magnitude  of  risks.  It  hides  the  effects  of  correlation  among  risks  with 
common  sources.  The  traditional  DoD  formulation  characterizes  risk  by  the  probability  of  the  event  and 
the  severity  of  the  consequence.  This  does  not  account  for  situations  where  the  severity  of  the 
consequence  depends  on  the  magnitude  and/or  timing  of  the  cause.  It  implicitly  assumes  that  the 
probabilities  of  future  events  and  their  consequences  can  be  reliably  estimated.  Other  fields  define  risk 
more  generally  and  take  a  broader  view,  e.g.,  the  International  Standards  Organization  (ISO)  defines  risk 
as  "the  effect  of  uncertainty  on  objectives."  In  addition  to  considering  specific  events,  other  fields 
employ  risk  indicators  to  assess  the  risk  exposure  of  the  program  and  alternative  decision  choices.  Risk 
indicators  are  essentially  leading  indicators  of  cost,  schedule  and  performance  (CSP)  slip  and 
uncertainty.  Other  fields  acknowledge  that  probabilities  are  not  meaningful  for  some  events  such  as 
choices  made  at  decision  points. 

Observation  #2:  Traditional  risk  management,  as  commonly  practiced  on  DoD  programs,  treats  each 
program  as  new  and  unique,  without  a  principled  approach  to  identify  potential  risk  areas  based  on  the 
program  organization  and  Work  Breakdown  Structure  (WBS),  and  without  being  informed  by  evidence 
from  past  similar  programs  regarding  the  characteristic  types  of  risk,  risk  sources,  and  mitigation 
strategies.  Other  approaches  to  risk  management  employ  Risk  Breakdown  Structures  (RBSs)  -  an 
organized  view  of  potential  risk  types,  sources  and  areas  affected  -  based  on  the  program  organization 
and  WBS,  and  accumulated  evidence  from  past  similar  programs.  Project  Failure  Modes  and  Effects 
Analysis  (P-FMEA)  and  other  structured  methods  are  used  to  develop  the  RBS. 

Observation  #3:  Traditional  risk  management,  as  commonly  practiced  on  DoD  programs,  does  not  treat 
program  decisions  as  risk  events.  A  decision  is  a  choice  from  among  alternatives,  or  a  value  assigned  to 
a  program  or  system  parameter.  The  outcomes  of  decisions  affect  the  expected  time,  cost  and 
performance,  and  affect  the  uncertainty  regarding  time,  cost  and  performance.  Risk  management 
should  provide  input  to  program  decision  making  regard  the  risk  impact  of  alternative  choices.  Evolving 
practice  in  Systems  Engineering  (SE)  in  some  elements  of  DoD  is  to  use  a  Decision  Breakdown  Structure 
(DBS)  as  the  organizing  principle  for  SE  analysis  and  support  to  program  management.  The  DBS  is  an 
organized  presentation  of  the  sequence  and  hierarchy  of  program  decisions  and  choices,  by  acquisition 
stage  and  aligned  with  the  project  development  artifacts  and  WBS.  The  DBS  is  derived  from  acquisition 
policy  and  guidance,  augmented  from  experience  on  past,  similar  programs. 

Observation  #4:  Traditional  DoD  risk  management  makes  limited  use  of  objective  indicators  of  program 
and  system  progress  and  status  to  assess  risk  exposure,  to  assess  the  risk  impact  of  decision  choices,  and 
to  alert  program  management  to  changes  in  risk  exposure  that  warrant  consideration  of  risk  mitigation. 
Other  domains  rely  extensively  on  risk  indicators  and  risk  estimating  relationships  (RERs)  to  monitor  and 
diagnose  program  risk  exposure,  and  to  set  "tripwires"  to  alert  program  management  to  an  elevated  risk 
condition  in  risk  tracking.  A  person's  cholesterol  level  is  an  indicator  for  the  risk  of  a  heart  attack. 
Physicians  have  developed  guidelines  for  the  combination  of  high-  and  low-density  lipoprotein  (HDL  and 
LDL)  cholesterol  levels.  The  physician  interprets  the  guidelines,  considering  other  patient  factors  such  as 
age,  gender,  family  history,  etc.,  to  recommend  mitigation.  Indicators  can  include  recent  performance 
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of  the  program,  analyses  of  program  artifacts,  and  Subject  Matter  Expert  (SME)  rating  scale 
assessments. 

Observation  #5:  Traditional  DoD  risk  management  does  not  employ  methods  to  assess  the  uncertainty 
or  confidence  intervals  of  the  risk  assessments,  or  use  such  information  to  make  informed  tradeoffs 
between  Type  I  and  Type  II  errors  in  deciding  whether  or  not  to  implement  a  risk  mitigation  (i.e.,  failure 
to  act  on  a  risk  versus  acting  unnecessarily).  Other  fields  employ  "group  dynamics"  methods  to  resolve 
miscommunication,  then  measure  then  degree  of  agreement  among  groups  of  stakeholders  and  SMEs 
to  assess  reliability  of  subjective  ratings.  When  data  from  previous  similar  programs  are  available, 
statistical  analysis  together  with  SME  ratings  are  combined  to  both  estimate  risk  and  the  uncertainty  in 
the  estimate.  Financial  underwriting  for  commercial  projects  and  system  reliability  analysis  have  both 
made  significant  advances  in  this  approach.  There  are  many  subtle  issues  in  determining  how  relevant 
one  program  is  to  another  for  extrapolation.  This  topic  is  not  a  priority  for  the  next  phase. 

Observation  #6:  Traditional  DoD  risk  management  does  not  collect  data  and  evidence  on  programs, 
agencies  and  contractors  in  a  rigorous  manner  and  centralized  repository,  with  predictive  analytics,  to 
support  and  inform  risk  assessment  and  confidence  assessment.  Other  domains  have  developed 
organizations  and  programs  to  collect  data  in  consistent  ways  across  programs,  clean  and  store  the  data, 
produce  analytics  methods  for  future  programs  to  make  quantitative,  evidence-informed  estimates  of 
risk  exposure  and  uncertainty  in  the  estimates.  There  are  many  complex  issues  in  collecting  data  across 
DoD  acquisition  programs.  This  topic  is  not  a  priority  for  the  next  phase. 


Decision  Breakdown  Structure 


The  Decision  Breakdown  Structure  (DBS)  approach  to  Systems  Engineering  for  DoD  system  acquisition 
programs  is  being  developed  by  US  Army  TARDEC  [1,  2],  and  has  been  used  in  other  industries  such  as 
construction  [3].  The  goal  of  the  DBS  is  to  provide  a  structure  to  organize  and  present  SE  analysis  - 
including  risk  management  -  in  a  context  that  is  directly  relevant  to  supporting  Program  Management 
decisions  and  activities.  It  is  being  developed  outside  of  this  project  to  be  the  organizing  framework  for 
Systems  Engineering  support  to  Program  Management.  The  DBS  is  based  on  policy  and  guidance, 
supplemented  with  knowledge  patterns  captured  from  actual  programs.  The  DBS  is  organized  by 
acquisition  stage,  program  artifacts,  formal  reviews,  and  high-level  WBS  structure.  Like  the  Work 
Breakdown  Structure  (WBS)  different  types  of  systems  may  have  different  DBS. 

The  DBS  identifies  the  decision  points  and  characteristic  choices  and  alternatives.  It  thus  provides  the 
framework  for  risk  management  to  assess  and  present  the  impacts  of  alternative  choices  on  the  risk 
exposure  of  the  program,  on  time  and  in  context  to  inform  Program  Management  decision  making.  The 
DBS  is  the  basis  for  Risk  Management  organized  to  around  and  targeted  to  support  specific  program 
decisions,  to  quantify  the  risk  exposure  impacts  before  the  decisions  are  made. 

The  DBS  is  the  organizing  principle  for  TARDEC's  Advanced  Systems  Engineering  Capability  (ASEC).  ASEC 
is  a  software  system  and  framework  that  links  SE  tools,  SE  products,  SE  data  sources,  and  historical 
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knowledge  patterns,  decision  patterns,  and  product  patterns  from  previous  programs.  ASEC  has  been 
used  for  over  a  year  on  Army  and  MCSC  programs.  It  is  on  schedule  to  be  used  on  additional  programs 
over  the  next  year.  It  is  under  continuous  development.  Capabilities  are  integrated  in  accordance  with 
the  needs  and  priorities  of  the  PEO  and  PM  customers. 

We  plan  to  work  with  TARDEC  and  their  DBS  framework,  not  to  duplicate  or  replace  it.  We  plan  to 
augment  the  DBS  with  a  corresponding  RBS  that  identifies  characteristic  risks  associated  with  the 
decision  points,  mitigation  strategies,  and  risk  warnings  and  indicators.  We  plan  to  augment  the  DBS  by 
providing  MTP  to  assess  the  risk  impact  of  decision  alternatives  and  choices.  We  plan  to  identify 
characteristic  risks  by  analysis  of  evidence  from  past  programs  and  review  of  related  studies. 


Risk  Breakdown  Structure 


The  Risk  Breakdown  Structure  (RBS)  complements  the  DBS  [4,  5,  6].  It  is  a  concept  that  has  gained 
prominence  in  the  past  ten  years.  The  RBS  is  a  tool  analogous  to  the  Work  Breakdown  Structure.  RBS  is 
provides  an  organized  prior  knowledge  of  the  types  and  sources  of  risk,  associated  indicators  and 
potential  mitigations.  The  goals  are  to  reduce  gaps  (overlooked  risks),  and  to  expedite  risk 
identification,  tracking  and  analysis.  In  the  DoD  systems  acquisition  context,  the  RBS  will  be  organized 
by  acquisition  stage,  program  management  decision  phase  or  artifact,  and  program  activity  or  WBS 
element.  The  RBS  can  also  include  organizational  dimensions  of  stakeholder  interest,  etc.  Figure  1 
presents  an  example  of  a  simple  RBS  taken  from  [4]. 
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Figure  1:  Example  Risk  Breakdown  Structure 


Risk  Indicators 


Risk  indicators  are  factors  that  indicate  the  type  and  magnitude  of  risk  exposure,  and  are  at  the  heart  of 
risk  estimating  relationships  to  compare  alternative  decision  choices  and  risk  tracking  warning  levels. 
The  risk  indicators  and  assessment  methods  are  expected  differ  by  acquisition  stage,  program  activity, 
and  system  type.  Risk  indicators  cover  planning,  execution,  and  engineering/technology  development 
and  integration.  The  risk  indicators  can  potentially  include  a  variety  of  types  of  data,  e.g.: 

(1)  Program  performance  trends,  e.g.,  the  rate  that  requirements  are  added,  deleted  or  changed, 
the  mean  and  standard  deviation  of  schedule  variance  for  completed  work  packages,  the 
proportion  of  tasks  started  late,  etc. 

(2)  Program  artifact  characteristics  current  state,  e.g.,  the  number  of  distinct  alternatives  being 
considered  in  the  AoA,  the  number  of  dependencies  in  the  Integrated  Master  Schedule  (IMS), 
etc. 

(3)  Program  assessments  (SME  ratings),  e.g.,  the  experience  of  the  contractor  with  engineering 
development  integrating  the  particular  technologies,  etc. 
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(4)  Historical  data  for  this  type  of  program  and  agent,  e.g.,  the  historical  cost-schedule  overrun 
work  packages  of  a  given  type  on  this  class  of  system 

The  distinction  between  "quantitative"  and  "qualitative"  data  is  obtuse,  and  not  necessarily  relevant.  A 
count  of  the  numbers  of  nodes  and  links  in  a  DODAF  SV-6  diagram  is  a  quantitative  measure,  but  the 
input  is  subjective  and  qualitative.  SME  ratings  and  questionnaire  responses  are  subjective  judgments, 
but  can  incorporate  intangibles,  and,  with  appropriate  anchoring  and  scaling,  can  provide  valuable  data. 
User  acceptance  is  a  "bottom  line"  measure  of  value,  but  is  subjective.  The  pragmatic  approach  is  to 
consider  all  inputs,  then  select  those  that  are  useful  and  reliable. 

Developing  an  appropriate  set  of  risk  indicators,  and  methods  to  assess  them,  is  central  to  the  approach. 
There  is  a  significant  body  of  sources  of  potential  risk  indicators  and  assessment  methods  to  draw  upon, 
e.g., 

(1)  "Leading  indicators"  of  program  success  and  uncertainty  (e.g.,  INCOSE  and  NDIA); 

(2)  Risk  indicators  sets  developed  by  research  institutes  to  compile  project  databases  and  forecast 
outcome  bias  and  dispersion  (e.g..  Helmsman,  PERIL,  Calleam,  and  RAND) 

(3)  Risk  indicators  that  could  be  derived  from  "Best  Practice"  guides  for  planning  and  execution 
(e.g.,  GAO,  DCMA,  and  Mitre); 

(4)  Risk  indicators  specifically  addressing  time  and  effort  to  integrate  technologies  (e.g.,  NASA  and 
TARDEC) 

We  plan  to  leverage  prior  research  and  development  of  program  risk  indicators.  In  the  next  phase  we 
plan  to  identify  unifying  themes,  and  adapt/formulate  practical  indicators  and  evaluation  methods.  In 
the  current  phase,  we  identified  just  over  a  dozen  sources  of  risk  indicators  or  "Best  Practices"  from 
which  indicators  could  be  derived  in  the  published  literature.  The  sources  are  briefly  summarized 
below. 

Systems  Engineering  Leading  Indicators  [7,  8].  The  leading  indicators  are  a  list  18  factors  and 
assessment  questions/criteria  that  have  been  found  to  be  useful  predictors  of  program  outcome  and/or 
variability.  The  specific  set  of  indicators  has  been  evolving. 

RAND  Risk  Assessment  Methodology  and  Tool  [9].  The  methodology  and  tool  is  organized  by  program 
review  -  Alternative  System  Review,  System  Requirements  Review,  etc.  At  each  review,  there  are  a  set 
of  program  maturity  questions  corresponding  to  the  review  exit  criteria.  The  tool  adds  fields  for  the 
importance  and  assessment  from  which  it  computes  a  score  and  weight,  which  are  aggregated  into 
numerical  assessments  of  completeness,  weight  and  risk.  The  tool  has  not  yet  been  validated  and  is 
released  only  as  a  draft.  The  tool  and  documentation  were  published  in  early  November,  2013. 

Helmsman  Institute  Risk  Indicators  [10,  11].  The  Helmsman  Institute,  in  collaboration  with  several 
Australian  Universities,  developed  and  refined  a  set  of  risk  indicators  and  assessment  procedures 
addressing  context,  experience,  requirements,  technical  and  project  management  issues,  with  a  set  of 
about  35  indicators.  The  indicators  have  been  evaluated  for  over  2,000  projects,  including  defense 
programs,  oilfield  development,  information  systems  and  finance.  The  project  also  collected  data  on 
cost  increase  and  schedule  slip,  and  eliminated  indicators  that  did  not  help  explain  outcome  variation. 
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While  the  Helmsman  database  may  not  be  relevant,  there  is  evidence  that  the  risk  indicators  are  both 
relevant  and  practical.  Detailed  description  of  the  indicators  and  assessment  methods  have  not  been 
reviewed,  but  the  Helmsman  Institute  has  agreed  to  make  them  available. 

Project  Experience  Risk  Library  (PERIL)  [12].  The  PERIL  project  developed  a  framework  of  25  risk 
indicators  in  three  categories  (scope,  resources  and  execution),  and  used  this  framework  to  collect  and 
analyze  schedule  slip  and  variability  on  over  800  Information  Technology  programs.  While  the  data  in 
the  database  may  not  be  relevant,  there  is  evidence  that  the  risk  indicators  are  practical  and  relevant  to 
schedule  slip  and  uncertainty. 

NASA  Advancement  Degree  of  Difficulty  [13,  14].  The  NASA  Advancement  Degree  of  Difficulty  (AD2)  is 
a  rating  system  that  complements  and  extends  the  Technical  Readiness  Level  (TRL)  rating  system.  The 
TRL  scale  addresses  only  the  current  level  of  technology  maturity,  not  the  difficulty  in  maturing  and 
integrating  the  technology.  The  nine  criteria  and  levels  of  the  AD2  are  intended  to  provide  insight  into 
the  time,  effort  and  uncertainty  in  fully  maturing  and  integrating  a  technology  or  subsystem. 

NASA  Cost  Uncertainty  Rating  Templates  [15].  The  cost  uncertainty  rating  questions  and  scale  in  the 
Cost  Uncertainty  Handbook  are  organized  by  project  phase,  and  are  intended  to  indicate  cost  estimating 
uncertainty  in  technology  maturation  and  integration.  The  provide  a  much  more  detailed  interrogation 
than  the  earlier  Advancement  Degree  of  Difficulty  rating. 

Integration  Readiness  Level  (IRL)  [16].  The  integration  readiness  level  framework  is  a  set  of  nine  criteria 
to  assess  the  readiness,  and  by  extension,  risk,  of  attempting  to  integrate  a  subsystem  or  technology. 
The  IRL  ratings  complement  TRL  and  MRL  (Manufacturing  Readiness  Level)  ratings.  IRL  ratings  have 
been  and  continue  to  be  used  by  TARDEC  and  other  agencies  as  an  input  to  risk  assessment. 

DCMA  14  Point  Schedule  Assessment  [17].  The  Defense  Contract  Management  Agency  (DCMA)  has 
published  14  quality  assessment  indicators  for  the  Integrated  Master  Schedule  (IMS).  The  IMS  is  a  key 
planning  artifact.  Deficiencies  in  planning  artifacts,  such  as  the  IMS,  are  likely  to  be  correlated  with 
increased  program  risk. 

GAO  Best  Practices  [18,  19,  20].  The  GAO  has  published  a  several  volumes  of  "Best  Practices"  for 
costing,  scheduling  and  system  planning.  These  documents  contain  explicit  and  implicit  indicators  of 
quality  and  reference  levels  to  achieve. 

Calleam  Risk  Indicators  and  RBS  [21].  Calleam  Consulting  (University  of  British  Columbia)  documented 
an  extensive  analysis  of  project  failure  modes  (the  RBS)  and  associated  risk  indicators.  The  practicality 
and  relevance  of  the  indicators  has  not  been  independently  assessed. 

AMSAA  Reliability  Scorecard  [22,  23].  The  Army  Material  Systems  Analysis  Agency  (AMSAA)  produced  a 
a  program  scorecard  with  approximately  25  indicators  organized  into  8  categories,  for  rating  scale 
assessment  to  estimate  eventual  system  reliability.  The  theoretical  relevance  is  that  program  factors 
that  produce  lower  levels  of  reliability  are  also  likely  to  be  associated  with  cost,  schedule  and 
performance  shortfall  and  uncertainty.  The  scorecard  has  been  in  use  in  the  Army  for  several  years,  and 
is  a  published  product. 
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Tables  1  and  2  present  example  lists  of  risk  indicators.  The  risk  indicators  in  Table  1  are  the 
NDIA/INCOSE  SE  leading  indicators  [7],  The  risk  indicators  in  Table  2  are  the  DCMA  scheduling  metrics 
[17],  Analysis  of  scope  and  overlap  of  the  indicators  from  available  sources  is  pending. 


1.  Requirements  Trends 

2.  System  Definition  Change  Backlog  Trends 

3.  Interface  Trends 

4.  Requirements  Validation  Trends 

5.  Requirements  Verification  Trends 

6.  Work  Product  Approval  Trends 

7.  Review  Action  Closure  Trends 

8.  Technology  Maturity  Trends 

9.  Risk  Exposure  Trends 

10.  Risk  Treatment  Trends 

11.  Systems  Engineering  Staffing  and  Skills  Trends 

12.  Process  Compliance  Trends 

13.  Technical  Measurement  Trends 

14.  Facility  and  Equipment  Availability 

15.  Defect  and  Error  Trends 

16.  System  Affordability  Trends 

17.  Architecture  Trends 

18.  Schedule  and  Cost  Pressure 

Table  1:  SE  Leading  Indicators 


1.  Logic 

2.  Leads 

3.  Lags 

4.  Relationship  Types 

5.  Flard  Constraints 

6.  Fligh  Float 

7.  Negative  Float 

8.  Fligh  Duration 

9.  Invalid  Dates 

10.  Resources 

11.  Missed  Tasks 

12.  Critical  Path  Test 

13.  Critical  Path  Length  Index 

14.  Baseline  Execution  Index 

Table  2:  DCMA  14-Point  Schedule 
Assessment  Metrics 
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Next  steps 


The  next  steps  are  to  select  an  initial  set  of  risk  indicators,  to  develop  a  RBS  for  a  suitable  stage  of 
acquisition,  then  work  with  the  TARDEC  end-user  co-developer  transition  partner  to  pilot  test,  assess, 
refine  and  transition  the  MPT. 
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